Today I’d like to talk about automated testing with iOS, and do some reviews as well as share my thoughts on various technologies used to do that. Firstly, I want to mention that in this post I will be talking specifically about regression and functional testing (i.e. going through the various screens of the app, clicking on things, making sure everything is working) of the UI and not performance testing (i.e. not trying to see how many users or actions the application can withstand even though testing that may require automating the UI). The difference is important because in performance testing we do not really care if things show up properly, just as long as they show up within a certain amount of time or show up at all.
Before we continue, you should make sure you are familiar with a few key terms I’m going to use in the rest of the article.
Hybrid App: An app that has both a native and web view (WebView / UIWebView) component.
UISpec: Apple’s API available internally to an iOS app to perform gestures and query for controls.
Types of Automated Testing Systems for iOS
There are several categories for how the automated testing systems work for iOS. I’ll break it down one by one.
This category contains tools like Calabash and Frank; essentially how it works is that you include a library in your app project which runs a web server with a REST API from your app. The reason I’m referring to this group as “Objective-C Framework” is because the framework itself is built in Objective-C and embedded into the iOS app (and iOS apps are written in Objective-C). This API allows some remote tool to connect to the app over the network and query for controls, issue gestures and commands, and consequently perform your automated tests. The actual interactions are based on a black-box Apple-made API called UISpec which is subject to change whenever Apple wants to update it, thus making these frameworks possibly break and malfunction if features are changed or removed (as happened between the update from iOS6 to iOS7).
There’s some other interesting ways to do the testing which rely on technologies like VNC and thus don’t fall into either category of the above.
- All interactions require traversing the full element hierarchy; that is you can’t simply say “tap the element with id foo” but you have to literally say thisWindow.thatScrollView.thatGridView.thisTile.thatButton[‘foo’] which makes tests quite brittle if you move elements around
- The generated macro code is really garbled and hard to read as it is automatically generated for you. You can go in and clean it up manually, but you still run into the issues above.
- Verification of elements for hybrid apps is incredibly difficult to do; when you record interactions on a webview it will resort to using coordinates for input only and completely ignore that the elements you’re tapping on may have better identifiers. Verification of the contents inside a webview is basically impossible except for finding if some text is contained in it.
Conclusion: Brittle, very macro-like and difficult to maintain. Thumbs down.
Calabash, KIF, & Frank
Calabash and Frank are basically the same (in fact, the Calabash project was inspired by Frank) so I will address them together. KIF is also here because like Calabash and Frank, it falls into the UISpec Objective-C framework category of automated testing systems for iOS (meaning it has more or less the same limitations). As of iOS7, Apple removed the functionality in UISpec that allowed simulating gestures like tap-and-hold, drag-and-drop, swipe, flick, etc. so right away the functionality of these technologies is quite limited as you can only tap on things programmatically. There’s not really much in the way of workarounds (sure there is a ‘scroll’ function that can be used similarly to swipe but it is not a full replacement as it requires your target to be a scroll view and won’t work in many other cases). In addition, because the framework is internal to the app, it cannot achieve many things you may want to simulate like locking/unlocking the device, rotating the device. On the plus side Calabash and Frank both work pretty well with hybrid apps in that they allow you to use CSS selectors to access elements inside the web-view.
KIF has an added disadvantage that it doesn’t seem to be maintained much anymore, and there doesn’t seem to be any active development going on with it.
Conclusion: Unless you’re running your app in iOS6 or have a very simple app that only has the need for ‘tap’ gestures and nothing else, this technology won’t help you. This is an unfortunate reality, but people are definitely moving away from calabash for this reason.
Zucchini is based on UIAutomation and doesn’t seem to have any workarounds regarding the UIAutomation lack of support for webviews as I outlined before. If you’re not developing a hybrid app, this may definitely be a good solution for you. Zucchini supports Gherkin statements and since it uses UIAutomation for the grunt work, it supports all those pesky gestures that Calabash and Frank don’t. A quick look at the ‘issues’ part of the project on Github suggests that this framework is still not exactly stable and has many issues. Issues like “Instruments hangs indefinitely between features on iOS7/Xcode5” are plentiful.
Conclusion: If you’re building a native (non-hybrid) app, this one may be a great solution. Otherwise, thumbs down.
Combines UISpec and the objective-c framework route with the UIAutomation route to try to harness the advantages of both. Allows identifying and querying elements based on ID’s instead of traversing the full element hierarchy like you would have to do in UIAutomation which is a big plus but it still relies on UIAutomation and thus inherits some of its problems and issues like “Scroll views are not tappable in iOS Simulator”, “Cannot drag scroll views in iOS7 Sim” and “Tests occasionally fail to launch in XCode5” (just based on the Github issues page). Personally, I think the idea of combining the framework and UIAutomation makes this framework a bit overcomplicated and introduces too many points of failure to be useful. It also doesn’t seem to have great support for hybrid apps, and doesn’t have support for Gherkin statements.
Conclusion: Seems to have way too many issues (I have not tried it out myself though) but your mileage may vary.
Appium & ios-driver
I’m putting these two together because they are very incredibly similar in how they work. So to give you a bit of understanding in case you’re not familiar with it, let’s backtrack to the world of browser testing. I’m talking real, desktop, full-fledged browsers like Chrome and FireFox. There’s a big demand for being able to test websites similarly to how we want to test iOS apps, and a standard called “Web Driver” was developed which is basically a way to control the browser via a REST API. There’s a full spec writeup here. Now, this is fantastic for browsers but what does it have to do with iOS testing you ask? Well, simply put, both Appium and ios-driver decided to implement an application that houses the REST API for webdriver, extends it with features relevant to iOS like swiping, tap-and-hold, etc. and then ties into instruments and UIAutomation to actually do all the grunt work. What you get is a completely language and technology agnostic way to write your tests as there’s webdriver bindings for pretty much every programming language you might want to work with. This means you can write your tests with ruby, java, node.js, or C# and they will work just the same.
Now on to the nitty-gritty part; how does it actually fare with iOS application testing? Well, let’s start off by saying I couldn’t even get ios-driver running (I got it to launch and load but it doesn’t actually do anything when I run my tests, not even start the simulator) so the rest of this part of the review will be solely about Appium. Appium looked very clean and worked really well right out of the box. It started the simulator with no problems, executed commands and did its thing. It even has support for physical device testing but I didn’t try that out myself. So where does it fall short? Hybrid apps. While ios-driver and Appium both claim to at least partially support webviews and hybrid app testing, I couldn’t actually get that work on Appium. Native controls responded fine, but inside a webview I could not actually simulate any kind of gesture (even plain tapping) as it had no effect even though it sent back a success message.
I also want to sidetrack here real fast to explain how appium and ios-driver actually achieve hybrid app support. This is interesting to consider for two reasons; one is that since they both rely in UIAutomation to do the heavy lifting, and given that UIAutomation is terrible at supporting web views, how do they manage? And secondly, how does the logistics of webdriver which is made for browsers (and not iOS apps) actually implement this? To answer the first part; the webviews on iOS are actually just safari running in an ‘isolated’ environment right? Well even Mobile Safari contains exposes the API for WebDriver! When enabled in the settings, appium can then connect to safari and issue the same webdriver commands to it directly, and thus have full awareness of the various elements in the webview even though UIAutomation does not. As for the second part, the logistics of how to implement that in webdriver – it’s quite ingenious! So when testing a website, your website may contain popups and other windows. The webdriver spec allows you to query for window handles and switch windows to be able to support that. in Appium and ios-driver, they simply make the webview return as a separate window – thus you can enter and leave the webview like you would a separate browser window and thus query it with the CSS selectors you want and still be able to query the native app with the iOS accessibility identifiers.
Couple other notes; I found the documentation for Appium quite lacking and ios-driver has a bazzillion issues listed on Github.
Conclusion: Appium is very promising, but I could not actually get the webview support to work even though it claims it supports it. The lack of documentation makes it really hard to track down what I can do (if anything) to fix that.
MobileCloud Automation ($$$)
I’d like to say straight-up that I did not actually try MobileCloud as it is a pay-for service/product and that was not an option for me. Some advantages based off their video and docs are that it allows testing on real, physical mobile devices that they control and it seems to support interacting outside of the app (e.g. flipping between the home screen pages). The actual tests look very macro-like (similar to UIAutomation) and thus I think it would be really difficult to maintain if you have a complicated app with a lot of different tests.
Conclusion: Very niche type of automated testing (macro style), and with a price tag.
These guys have a free trial available on their site, which I have not tried myself, but I have read a bunch of their documentation and watched some of the videos to know a general sense of how it works. It is a really interesting take on automated testing and it works in a completely different way from all the other technologies listed here. The way it works is that eggPlant supplies you with a custom built VNC app (if you’re not familiar with VNC, think Remote Desktop but more of an open standard) that allows a remote connection to take control of your iPad or other device. The VNC server (called eggPlant Mobile / eggOn) is not a part of the free trial; only the actual testing software is. They then have an application for testing that connects to said VNC server and uses OCR to pretty much do everything. You supply a bunch of cut-outs of images (say a picture of a button you want to press, or a picture you want to verify that it is on the screen) which the tool supposedly makes easy for you to do and then it goes to do its thing. This means that it is completely independent of the issues faced by UISpec-style frameworks like Calabash as well as the limitations of UIAutomation-style frameworks like Appium. Very unique solution.
As I said, I have not actually tried it out myself but I could definitely see some drawbacks just by knowing the limitations of modern-day OCR technology. The issues would include things like:
- If the app changes visually in any way, the tests must be updated (even if its just a color change of a button)
- Viewing in another language would instantly break the test as the characters won’t match up
- Using weird non-serif fonts would probably greatly impact the OCR character recognition for when you’re looking for text
Conclusion: Seems like a really great idea, and would probably work well with very graphical apps (like games) but probably not for text-heavy applications.
From my experience, automated testing for iOS is just not worth it if you have a complicated hybrid app as there’s no one technology that does this kind of testing well. However if you have a fully-native app I would probably recommend Appium as it was the easiest to set up, work with, and has many nice features like supporting physical devices.
Inspiration & Shout-outs
I have pieced together a lot of the information above from trying to use these technologies firsthand, but also from some other blog posts which I found really useful. That being said, I’d like to give a quick shout out to the authors of the following posts as it really helped me a lot in doing my own research, though I wanted to elaborate on their thoughts so I made this post.