Automated Testing of iOS Apps

Today I’d like to talk about automated testing with iOS, and do some reviews as well as share my thoughts on various technologies used to do that. Firstly, I want to mention that in this post I will be talking specifically about regression and functional testing (i.e. going through the various screens of the app, clicking on things, making sure everything is working) of the UI and not performance testing (i.e. not trying to see how many users or actions the application can withstand even though testing that may require automating the UI). The difference is important because in performance testing we do not really care if things show up properly, just as long as they show up within a certain amount of time or show up at all.

Definitions

Before we continue, you should make sure you are familiar with a few key terms I’m going to use in the rest of the article.

Hybrid App: An app that has both a native and web view (WebView / UIWebView) component.

UISpec: Apple’s API available internally to an iOS app to perform gestures and query for controls.

Types of Automated Testing Systems for iOS

There are several categories for how the automated testing systems work for iOS. I’ll break it down one by one.

Objective-C Framework

This category contains tools like Calabash and Frank; essentially how it works is that you include a library in your app project which runs a web server with a REST API from your app.  The reason I’m referring to this group as “Objective-C Framework” is because the framework itself is built in Objective-C and embedded into the iOS app (and iOS apps are written in Objective-C). This API allows some remote tool to connect to the app over the network and query for controls, issue gestures and commands, and consequently perform your automated tests. The actual interactions are based on a black-box Apple-made API called UISpec which is subject to change whenever Apple wants to update it, thus making these frameworks possibly break and malfunction if features are changed or removed (as happened between the update from iOS6 to iOS7).

calabash-ios

Instruments

Instruments is a part of XCode which contains a tool called “UIAutomation”. This is a built-into-XCode way of performing UI tests via javascript code. Many other systems for automated testing tie into this functionality and essentially act as an abstraction layer between your tests and the UIAutomation. Once again, since UIAutomation itself is a closed source black-box from Apple (black box in the sense that Instruments itself is closed source so you can’t extend it yourself or see how it works behind the scenes), changes to it may break the framework on which you rely to do your tests.

instruments-selection

Other

There’s some other interesting ways to do the testing which rely on technologies like VNC and thus don’t fall into either category of the above.

UIAutomation

uiautomation

First let’s address the biggest and most promising technology for doing your automated tests; Apple’s own UIAutomation via Instruments. UIAutomation is available within XCode and it allows you to record a macro-like sequence of steps which are outputted in JavaScript and then play those steps back. You can interact with a logger, assert expectations, and more. It’s also made by Apple itself so it has a huge one-up on any other technology in that it is made by the same people who made iOS. Sounds great right? Unfortunately, unless you’re developing some really basic app, you’re out of luck with this tool. It has many critical drawbacks including:

  • It’s difficult (but possible!) to work with multiple javascript files being included in one test (if you wanted to build up a library for reusable code)
  • All interactions require traversing the full element hierarchy; that is you can’t simply say “tap the element with id foo” but you have to literally say thisWindow.thatScrollView.thatGridView.thisTile.thatButton[‘foo’] which makes tests quite brittle if you move elements around
  • The generated macro code is really garbled and hard to read as it is automatically generated for you. You can go in and clean it up manually, but you still run into the issues above.
  • Verification of elements for hybrid apps is incredibly difficult to do; when you record interactions on a webview it will resort to using coordinates for input only and completely ignore that the elements you’re tapping on may have better identifiers. Verification of the contents inside a webview is basically impossible except for finding if some text is contained in it.

Conclusion: Brittle, very macro-like and difficult to maintain. Thumbs down.

Calabash, KIF, & Frank

Links: https://github.com/calabash/calabash-ios and https://github.com/kif-framework/KIF and http://www.testingwithfrank.com/

Calabash and Frank are basically the same (in fact, the Calabash project was inspired by Frank) so I will address them together. KIF is also here because like Calabash and Frank, it falls into the UISpec Objective-C framework category of automated testing systems for iOS (meaning it has more or less the same limitations). As of iOS7, Apple removed the functionality in UISpec that allowed simulating gestures like tap-and-hold, drag-and-drop, swipe, flick, etc. so right away the functionality of these technologies is quite limited as you can only tap on things programmatically. There’s not really much in the way of workarounds (sure there is a ‘scroll’ function that can be used similarly to swipe but it is not a full replacement as it requires your target to be a scroll view and won’t work in many other cases). In addition, because the framework is internal to the app, it cannot achieve many things you may want to simulate like locking/unlocking the device, rotating the device. On the plus side Calabash and Frank both work pretty well with hybrid apps in that they allow you to use CSS selectors to access elements inside the web-view.

KIF has an added disadvantage that it doesn’t seem to be maintained much anymore, and there doesn’t seem to be any active development going on with it.

Conclusion: Unless you’re running your app in iOS6 or have a very simple app that only has the need for ‘tap’ gestures and nothing else, this technology won’t help you. This is an unfortunate reality, but people are definitely moving away from calabash for this reason.

Zucchini Framework

Link: http://www.zucchiniframework.org/

Zucchini is based on UIAutomation and doesn’t seem to have any workarounds regarding the UIAutomation lack of support for webviews as I outlined before. If you’re not developing a hybrid app, this may definitely be a good solution for you. Zucchini supports Gherkin statements and since it uses UIAutomation for the grunt work, it supports all those pesky gestures that Calabash and Frank don’t. A quick look at the ‘issues’ part of the project on Github suggests that this framework is still not exactly stable and has many issues. Issues like “Instruments hangs indefinitely between features on iOS7/Xcode5” are plentiful.

Conclusion: If you’re building a native (non-hybrid) app, this one may be a great solution. Otherwise, thumbs down.

Subliminal

Link: https://github.com/inkling/Subliminal

Combines UISpec and the objective-c framework route with the UIAutomation route to try to harness the advantages of both. Allows identifying and querying elements based on ID’s instead of traversing the full element hierarchy like you would have to do in UIAutomation which is a big plus but it still relies on UIAutomation and thus inherits some of its problems and issues like “Scroll views are not tappable in iOS Simulator”, “Cannot drag scroll views in iOS7 Sim” and “Tests occasionally fail to launch in XCode5” (just based on the Github issues page). Personally, I think the idea of combining the framework and UIAutomation makes this framework a bit overcomplicated and introduces too many points of failure to be useful. It also doesn’t seem to have great support for hybrid apps, and doesn’t have support for Gherkin statements.

Conclusion: Seems to have way too many issues (I have not tried it out myself though) but your mileage may vary.

Appium & ios-driver

Links: http://appium.io/ and http://ios-driver.github.io/ios-driver/index.html

I’m putting these two together because they are very incredibly similar in how they work. So to give you a bit of understanding in case you’re not familiar with it, let’s backtrack to the world of browser testing. I’m talking real, desktop, full-fledged browsers like Chrome and FireFox. There’s a big demand for being able to test websites similarly to how we want to test iOS apps, and a standard called “Web Driver” was developed which is basically a way to control the browser via a REST API. There’s a full spec writeup here. Now, this is fantastic for browsers but what does it have to do with iOS testing you ask? Well, simply put, both Appium and ios-driver decided to implement an application that houses the REST API for webdriver, extends it with features relevant to iOS like swiping, tap-and-hold, etc. and then ties into instruments and UIAutomation to actually do all the grunt work. What you get is a completely language and technology agnostic way to write your tests as there’s webdriver bindings for pretty much every programming language you might want to work with. This means you can write your tests with ruby, java, node.js, or C# and they will work just the same.

Now on to the nitty-gritty part; how does it actually fare with iOS application testing? Well, let’s start off by saying I couldn’t even get ios-driver running (I got it to launch and load but it doesn’t actually do anything when I run my tests, not even start the simulator) so the rest of this part of the review will be solely about Appium. Appium looked very clean and worked really well right out of the box. It started the simulator with no problems, executed commands and did its thing. It even has support for physical device testing but I didn’t try that out myself. So where does it fall short? Hybrid apps. While ios-driver and Appium both claim to at least partially support webviews and hybrid app testing, I couldn’t actually get that work on Appium. Native controls responded fine, but inside a webview I could not actually simulate any kind of gesture (even plain tapping) as it had no effect even though it sent back a success message.

I also want to sidetrack here real fast to explain how appium and ios-driver actually achieve hybrid app support. This is interesting to consider for two reasons; one is that since they both rely in UIAutomation to do the heavy lifting, and given that UIAutomation is terrible at supporting web views, how do they manage? And secondly, how does the logistics of webdriver which is made for browsers (and not iOS apps) actually implement this? To answer the first part; the webviews on iOS are actually just safari running in an ‘isolated’ environment right? Well even Mobile Safari contains exposes the API for WebDriver! When enabled in the settings, appium can then connect to safari and issue the same webdriver commands to it directly, and thus have full awareness of the various elements in the webview even though UIAutomation does not. As for the second part, the logistics of how to implement that in webdriver – it’s quite ingenious! So when testing a website, your website may contain popups and other windows. The webdriver spec allows you to query for window handles and switch windows to be able to support that. in Appium and ios-driver, they simply make the webview return as a separate window – thus you can enter and leave the webview like you would a separate browser window and thus query it with the CSS selectors you want and still be able to query the native app with the iOS accessibility identifiers.

Couple other notes; I found the documentation for Appium quite lacking and ios-driver has a bazzillion issues listed on Github.

Conclusion: Appium is very promising, but I could not actually get the webview support to work even though it claims it supports it. The lack of documentation makes it really hard to track down what I can do (if anything) to fix that.

MobileCloud Automation ($$$)

Link: http://www.perfectomobile.com/Products/MobileCloud_Automation

I’d like to say straight-up that I did not actually try MobileCloud as it is a pay-for service/product and that was not an option for me. Some advantages based off their video and docs are that it allows testing on real, physical mobile devices that they control and it seems to support interacting outside of the app (e.g. flipping between the home screen pages). The actual tests look very macro-like (similar to UIAutomation) and thus I think it would be really difficult to maintain if you have a complicated app with a lot of different tests.

Conclusion: Very niche type of automated testing (macro style), and with a price tag.

eggPlant ($$$)

Link: http://www.testplant.com/eggPlant

These guys have a free trial available on their site, which I have not tried myself, but I have read a bunch of their documentation and watched some of the videos to know a general sense of how it works. It is a really interesting take on automated testing and it works in a completely different way from all the other technologies listed here. The way it works is that eggPlant supplies you with a custom built VNC app (if you’re not familiar with VNC, think Remote Desktop but more of an open standard) that allows a remote connection to take control of your iPad or other device. The VNC server (called eggPlant Mobile / eggOn) is not a part of the free trial; only the actual testing software is. They then have an application for testing that connects to said VNC server and uses OCR to pretty much do everything. You supply a bunch of cut-outs of images (say a picture of a button you want to press, or a picture you want to verify that it is on the screen) which the tool supposedly makes easy for you to do and then it goes to do its thing. This means that it is completely independent of the issues faced by UISpec-style frameworks like Calabash as well as the limitations of UIAutomation-style frameworks like Appium. Very unique solution.

As I said, I have not actually tried it out myself but I could definitely see some drawbacks just by knowing the limitations of modern-day OCR technology. The issues would include things like:

  • If the app changes visually in any way, the tests must be updated (even if its just a color change of a button)
  • Viewing in another language would instantly break the test as the characters won’t match up
  • Using weird non-serif fonts would probably greatly impact the OCR character recognition for when you’re looking for text

Conclusion: Seems like a really great idea, and would probably work well with very graphical apps (like games) but probably not for text-heavy applications.

Conclusion

From my experience, automated testing for iOS is just not worth it if you have a complicated hybrid app as there’s no one technology that does this kind of testing well. However if you have a fully-native app I would probably recommend Appium as it was the easiest to set up, work with, and has many nice features like supporting physical devices.

Inspiration & Shout-outs

I have pieced together a lot of the information above from trying to use these technologies firsthand, but also from some other blog posts which I found really useful. That being said, I’d like to give a quick shout out to the authors of the following posts as it really helped me a lot in doing my own research, though I wanted to elaborate on their thoughts so I made this post.

6 Comments

Vall says:

Excellent Article. I really loved the way you reviewed.

I have a requirement where in a UI is same between an IOS native app and desktop website, I already have scripts created in QTP for desktop web, but what i am looking for is a way, where in i can maintain same set of scripts for both IOS app and web app

Jeffrey Wear says:

Hi Petro, I’m the maintainer of Subliminal. I’d like to say that this is a very comprehensive roundup and you are quite right to say that by relying on UIAutomation we inherit some of its issues.

However, there are very few issues that we cannot work around by having access to the application. Where UIAutomation returns incorrect results (like saying that a view is visible when it really isn’t), Subliminal can examine the app directly; where UIAutomation cannot drag scroll views, Subliminal can manipulate them directly.

UIAutomation-based frameworks like Appium with no in-app component are also subject to these bugs, but cannot work around them.

Why use UIAutomation at all, you might ask? Well, as you mentioned in the Calabash, KIF, and Frank section, frameworks that are _only_ in the app cannot rotate the device, etc. Using UIAutomation lets us simulate the widest range of user interactions.

Subliminal should have no problems testing hybrid apps. As you note in the UIAutomation section, Instruments ignores web elements’ identifiers when recording, but those identifiers are still there, and can be read by Subliminal just like those of native elements.

Lastly, I confess I’m not familiar with the UISpec framework but that is because Subliminal does not use any such framework. 🙂 Our Objective-C APIs are entirely of our own design, and for the most part there is a clean division between the work done by them and UIAutomation: using Objective-C we search within the view hierarchy for the element to be manipulated; and then we direct UIAutomation to actually do the manipulation.

* Petro says:

Hi Jeffrey, thanks for your response! Really cool to have someone from one of the projects I reviewed directly reach out to me like this. I definitely agree with your points, and hope your comment helps others viewing this post but as for me I have since moved on to developing for a Windows Store app so I am currently not working with any of the aforementioned technologies anymore. Thanks again for taking the time to share your thoughts! 🙂

Cindy says:

Hi Petro,

Excellent article. Thanks for writing it!

I’m currently writing automated tests for an app that is supported on iOS and Android, and I was wondering if your recommendation for an iOS automated testing system has changed since the time that this article was written. At the time of writing you recommended Appium, but I constantly run into problems with it when I’m using it for iOS. (It works great for Android!) There is an active community for Appium but even they don’t have all of the answers to problems that I’ve had, and I find myself blocked for weeks at a time on issues that should be trivial, like scrolling.

Just curious whether your opinion on this has changed since February of 2014.

Thank you!

* Petro says:

Hi Cindy,
Thanks for your comment and sorry for the late response; I haven’t touched iOS automation since writing the article so unfortunately I can’t recommend anything at this time. I would assume that the official UIAutomation tool that Apple provides has improved since its time of release and maybe that is worth a shot? Good luck!

Petro.

Jack Martin says:

Very nice, i like the way you explained. I also wrote something on similar lines on Challenges Faced in Automation Testing. Hope you would like it – http://bit.ly/1O9SwWd

Leave a Reply

Your email address will not be published.