Kiwi is for unit testing, KIF is for integration testing. They don't do the same thing (nor try to). Since you said you're just starting to get into it, here's what that means:
Integration tests simulate a user using your app - tapping on controls, manipulating views, typing in text, etc (see the video in the blog post). As these steps run you have simple tests to make sure the app is behaving as you expect it to.
Unit tests exercise more discrete things like model objects, controller logic, validations. Tons of small tests like "when we add these money values together, is the sum what we expect?".
At Square we use KIF for integration tests and GHUnit for unit tests. We run both on a continuous integration system (CruiseControl.rb) that runs the tests whenever we push new code.
The best thing I got out of this is that I discovered Square's GitHub account... with an astonishing 65 repositories. I already held Square in high regard; this just icing on the cake.
Semi-related, does anyone perform automated unit test deployment, execution, and reporting on iOS devices? If so, how do you do it?
At my current company we have a rather large automated test infrastructure that builds and runs ~50 or so individual libraries and their associated unit test programs on around 20 different configurations for 10 different platforms. Most of these platforms have a way of doing an unattended deployment of our test programs (and associated data) to the device, execute the programs, and retrieve the output.
This isn't working on iOS though because of its dependency to Xcode. We've tried scripting Xcode with AppleScript but this is fragile and routinely breaks.
Yup. I've setup a Jenkins server that uses a combination of xcodebuild and xcrun. The process:
- clones the repo from git
- builds with xcodebuild
- signs/provisions with xcrun
- creates a manifest file for over-the-air installs
- copies the build to a server where users can install
- saves off the symbol file for our crash server so it can symbolicate crashes
- sends emails
It would run unit tests and integration tests if we had them. :)
Thanks for the xcrun hint! Does that still handle the application/data deployment and execution, or does it just automate the packaging? All the information I can gather about doing OTA deployment points to still requiring the user to initiate the download, and then run it. I'd like to avoid that if possible because of the sheer number of libraries we need to test.
1. You can't automate running of the UIAutomation tests. Square really wanted this for continuous integration.
2. Writing the UIAutomation scripts is really tedious (though
it's much better in iOS [redacted]).
3. The UIAutomation tests ran very slowly.
4. Having to write all the tests in JavaScript isn't very flexible.
That said, just like UIAutomation, KIF stands on the shoulders of the great accessibility features in iOS. In many ways KIF is like an Objective-C version of UIAutomation.
Our iOS team at Square is more comfortable and efficient
writing tests (and test libraries) in Objective-C than JavaScript. We actually wouldn't gain much from the 'scripting language'-ness of JavaScript, and the additional language and tool (Instruments) would make writing tests more complicated and painful.
Nice! I've been looking for some proper iOS functional testing for a weekend project. Along with WaxSim this feels like enough plumbing to roll into a Jenkins plugin.
Indeed, accessibility labels are the only way to reference views. Granted, your tests are running within your process, so they sky is pretty much the limit.
We're still considering the best way to implement gestures. There's some rudimentary support, but it's incomplete right now. Your input is welcome!
Also, keep in mind that many views just "do the right thing" as far as accessibility goes. Buttons for example will use their title as the default accessibility label if you don't set one explicitly.
It should report errors automatically. For the most part, it will detect problems automatically because a step will fail to execute. The reason is reported in the log that KIF creates. For example, if you had steps to
(1) enter the user name
(2) enter the password
(3) click "sign in"
(4) hit the refresh button
If the login failed for some reason, then step (4) would fail because it wouldn't be able to find the refresh button.
You can also add explicit verification steps, such as "wait for the view with the accessibility label 'Refresh'" if you want further validation.
Seriously, unit testing in iOS is not great, and this kind of simulation is exactly what we need. Regular unit tests are great for model-layer stuff and data-processing code, but they will not help with most real-world problems that occur.
"Does my login flow still work?" and "Can I still use every feature of my app successfully?" is the kind of thing you need to know after committing code.
--
[1]: http://corner.squareup.com/2011/07/ios-integration-testing.h...
[2]: http://http://www.kiwi-lib.info/