UI testing iOS apps for humans

No one likes to be blamed for the mistakes. This is especially true for software engineers. Over the years they came up with countless ideas which make software development less fragile and error-prone. It is safe to say that ever growing number of precaution steps taken to maximize the quality of the product is what defines the best developers in the business.

In today's article, I will share a great and relatively easy way to verify whether an app works and looks as expected from the end user's perspective. We will learn how to automatically prove that all the features work and that the app's looks were not broken during the development.

Motivation

Let's try to answer an example question to show what UI testing is and what are the motivations behind it.

Do we really need UI tests? If so, why? We already have unit tests...

UI testing serves a completely different purpose than unit testing. UI testing focuses heavily on the end-user. It verifies that the presentation is correct and that app's features are reachable and render correct output.

In comparison, unit testing focuses solely on what is visible to developers. It aims to prove that small chunks of source code work as intended and also verify whether their API is good enough. This means that unit and UI tests are complementary, not mutually exclusive.

Unfortunately, UI tests are commonly downplayed compared to unit tests^[1] due to several reasons:

• They have a significantly slower feedback loop.
• They might not be 100% reliable. UI testing is commonly considered to be fragile and associated with a lot of false positives.
• UI tests are viewed as painful to implement (cumbersome and repetitive code).

With that many issues, is UI testing even worth considering?

In my opinion, yes. UI testing is critical in customer-facing environments, which mobile apps market certainly represents. All in all, UI testing proves that the application works from your prospect's perspective.

Fortunately for us, iOS devs, UI testing is completely different from what it used to be. During the last couple of months, a lot of great 3rd party tools emerged that make UI testing fun. This article will briefly explore these tools and show how to use them for maximum productivity.

Xcode's UI testing

Before analyzing 3rd party tools, let's briefly examine a built-in Xcode solution for testing user interfaces - XCUITest.

XCUITest is a black-box testing utility. This means there is no way to communicate with application sources during testing. XCUITest requires creating a separate testing target. It exposes a special set of APIs dedicated for querying UI elements. Example test might look as follows:

import XCTest

class ExampleUITest: XCTestCase {
  override func setUp() {
    super.setUp()
    continueAfterFailure = false
    XCUIApplication().launch()
  }
    
  func testDispatchIsSuccessfullyInvoked() {
    let application = XCUIApplication()
    application.tables.cells.staticTexts["A Cell Text"].tap()

    let hasFinishedLabel = application.staticTexts["Dispatch has finished!"]
    let predicate = NSPredicate(format: "exists == 1")

    expectation(for: predicate, evaluatedWith: hasFinishedLabel, handler: nil)
    waitForExpectations(timeout: 5.0, handler: nil)
  }  
}

Moreover, XCUITest offers test recording utility, which is a code generator based on user interactions. The feature works quite nicely, but it lacks the ability to specify assertions.

Unfortunately, XCUITest has a couple of really serious pain points:

1. Every asynchronous action has to be verified using expectations (as shown in the example code). The testing code explicitly waits for a condition to be met with a specified timeout. This approach has some serious issues:

   • writing cumbersome code using aging, string-based predicates API,
   • too short timeouts may cause unexpected failures,
   • further changes to source code (like chaining more asynchronous actions or converting a synchronous action to an asynchronous one) may break the test when it shouldn't.

2. There is no easy way to inject a test specific behavior. Let's suppose you need to display some specific UI elements for French users only. How do you inject French data? You need to pass launch arguments and parse them inside main application source code.

3. The tests seem to be really slow. Two separate app containers are deployed to a simulator:

   • the main application,
   • the test application that executes automated script on the main application.

   Deployment takes time, the app switching takes time and the navigation state is lost between tests.

4. There is no way to assert view layouts.

The issues listed above make XCUITest completely unsuitable for writing lightweight, maintainable UI tests during the development process.

Fortunately, testing the views' layout is easier than you might think. It can be done in no time with the help of snapshot tests.

Snapshot testing

Snapshot testing is a technique in which two rendered views are compared pixel by pixel. The test fails if an examined snapshot differs from the reference one.

Snapshot testing is a process supervised by a developer:

1. Firstly, a correct implementation of the view is created^[2].
2. Secondly, the developer asks framework to take a reference snapshot of the view.
3. From now on, every test pass validates a target view against the reference image.

The easiest option to get snapshot testing for iOS applications is to use FBSnapshotTestCase. When using Cocoapods, integrating the framework is as easy as adding a single dependency to the test target:

target 'UnitTests' do
  use_frameworks!
  pod 'FBSnapshotTestCase', '~> 2.0'
end

and running pod install. Next, one has to define FB_REFERENCE_IMAGE_DIR and IMAGE_DIFF_DIR environment variables for the test target. This process is well documented in FBSnapshotTestCase readme.

Let's define our first snapshot test:

import FBSnapshotTestCase

class ViewControllerTests: FBSnapshotTestCase {
  override func setUp() {
    super.setUp()
    recordMode = true
  }

  func testViewControllersViewMatchesSnapshot() {
    let viewController = ViewController(nibName: nil, bundle: nil)
    viewController.view.frame = UIScreen.main.bounds

    FBSnapshotVerifyView(viewController.view)
  }
}

Notice how we define recordMode = true to create a reference image. If we run the tests now, they will fail:

As error message states, this is the expected behavior. We should disable the recording now. This can be done by setting recordMode = false. The tests will pass now.

If at any point in the future the view gets broken:

• unit tests will fail,
• an image containing the visual diff will be generated for you.

The image is especially useful when trying to figure out what went wrong. Let's see the example:

On the left-hand side, the expected view is presented. On the right-hand side - received view. In the middle, there is a visual diff between the two.

Reference images are an integral part of the snapshot test. It means that you have to commit them to the repository alongside the test sources.

Device agnostic snapshots

As you might have noticed, in our snapshot test we have set view's frame to screen size using:

viewController.view.frame = UIScreen.main.bounds

Obviously, such test will fail when executed on a device with different point screen size and/or pixel per point density. To fix that issue, you can tell FBSnapshotTestCase to record device agnostic snapshots. This means that the library will record and verify a single snapshot per <test, device type> pair.

To use device agnostic snapshots, simply add isDeviceAgnostic = true line to the setUp method of the snapshot test case:

class ViewControllerTests: FBSnapshotTestCase {
  override func setUp() {
    super.setUp()
    recordMode = true
    isDeviceAgnostic = true
  }
}

Snapshot testing with Quick

Up until now, we have used FBSnapshotTestCase base class instead of XCTestCase for snapshot testing. What if we would like to use snapshot tests with the Quick framework?

Fortunately, there is an already implemented Nimble matcher for snapshot testing. It is called Nimble-Snapshots. The example spec is presented below:

import Nimble
import Nimble_Snapshots
import Quick

class ViewControllerSpec: QuickSpec {
  override func spec() {
    describe("ViewController") {
      var sut: ViewController!

      beforeEach {
        sut = ViewController(nibName: nil, bundle: nil)
      }

      afterEach {
        sut = nil
      }

      describe("view") {
        it("has a valid snapshot") {
          sut.view.frame = UIScreen.main.bounds

          expect(sut.view).to(haveValidSnapshot())
          // to record the snapshot use:
          // expect(sut.view).to(recordSnapshot())
        }
      }
    }
  }
}

The matcher works exactly the same way as FBSnapshotVerifyView counterpart does. It is possible to use device agnostic snapshots, too. In order to do that replace the matchers with:

haveValidDeviceAgnosticSnapshot()    // for validation
recordDeviceAgnosticSnapshot()       // for recording

What about non-static views?

Snapshot testing is great, but how do you use it with networking, animations or user-interactions? Let's suppose we have an animated detail controller, as shown below:

The animation is triggered in viewWillAppear: method of containing view controller:

class AnimatedRectangleViewController: UIViewController {
  override func loadView() {
    view = AnimatedRectangleView()
  }

  override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    animatedRectangleView.playAnimation()
  }

  private var animatedRectangleView: AnimatedRectangleView {
    guard let view = view as? AnimatedRectangleView else { fatalError() }
    return view
  }
}

class AnimatedRectangleView: UIView {
  private let animationHasFinished = UILabel(frame: .zero)
  private var centerXConstraint: NSLayoutConstraint?
  private var centerYConstraint: NSLayoutConstraint?

  // subviews and constraints are set up in initializer

  func playAnimation() {
    centerXConstraint?.constant = 15
    centerYConstraint?.constant = 50

    UIView.animate(withDuration: 2.5, animations: {
      self.layoutIfNeeded()
    }, completion: { _ in
      self.animationHasFinished.isHidden = false
    })
  }
}

Let's try to implement a naive snapshot testing method:

import FBSnapshotTestCase

class AnimatedRectangleViewControllerTests: FBSnapshotTestCase {
  override func setUp() {
    super.setUp()
    recordMode = false
  }

  func testViewControllersViewMatchesSnapshot() {
    let viewController = AnimatedRectangleViewController()
    viewController.view.frame = UIScreen.main.bounds

    FBSnapshotVerifyView(viewController.view)
  }  
}

After recording the initial snapshot the test passes. But what does it check? The question can be answered by examining ReferenceImages directory:

Obviously, the snapshot test doesn't assert too much. It only tests initial layout, completely ignoring the animation. No matter how badly the animation gets screwed, the test is guaranteed to pass.

The brute force approach to fixing this problem is to change the view controller's API. We could either introduce the method that skips the animation or that invokes a completion block when the animation is finished. However, adding a new feature that is used only for testing purposes is not the brightest idea.

Functional tests

To assert the animations' results without modifying sources, a functional testing framework can be used. Having already eliminated XCUITest as a not viable option, I had to browse through available third parties. The one that really stands out for me is EarlGrey - the red-hot iOS UI testing framework from Google. What makes it different than other functional testing frameworks is the approach:

• White-box testing. By allowing to interact with the application code, one can use stubbing and mocking to prove correctness.
• No time-based and condition-based wait clauses. EarlGrey automatically synchronizes with network calls, animations, operation queues and delayed dispatches so that it can automatically detect a stable view state.

Quick start with EarlGrey is covered in installation & running docs, so I will omit the setup and jump straight to the testing code. Let's analize the colors example again:

Step zero is to make cells with animated colors distinguishable. We'll map their names to Animated {colorName}.

Our goal is to write a test that triggers the Animated Cyan cell tap, waits for the animation to complete and asserts whether Animation has finished label is visible:

import EarlGrey

class AnimatedRectangleFeatureTests: XCTestCase {
  func testRectangleShowsALabelAfterAnimationIsFinished() {
    EarlGrey.select(elementWithMatcher: grey_text("Animated Cyan"))
      .using(searchAction: grey_scrollInDirection(.down, 50.0),
             onElementWithMatcher: grey_kindOfClass(UITableView.self))
      .perform(grey_tap())
      .assert(grey_sufficientlyVisible())

    // view controller transition and animation takes place here

    EarlGrey.select(elementWithMatcher: grey_text("Animation has finished!"))
      .assert(grey_sufficientlyVisible())
  }
}

Let's analyze the first EarlGrey function call chain:

1. .select(elementWithMatcher: grey_text("Animated Cyan")) - we ask EarlGrey to find any UI element which displays Animated Cyan text. By default, grey_text matcher takes UILabel's, UITextField's and UITextView's into account.
2. .using(searchAction: grey_scrollInDirection(.down, 50.0), onElementWithMatcher: grey_kindOfClass(UITableView.self)) - since we look for a cell inside the table view, we might need to scroll down to find it. The clause tells EarlGrey to search the given UI element by scrolling down the first table view that can be found on the screen.
3. .perform(grey_tap()) - invokes tap action on the matched element.
4. .assert(grey_sufficientlyVisible()) - make sure that the element is visible. This call might seem useless, but calling .assert activates synchronization chain. We want this behavior, because we want to synchronize on next animation finished event.

Here is where the EarlGrey magic happens. Upon the tap:

1. A new view controller is pushed onto the navigation stack.
2. The animation is triggered.

Using typical UI testing framework, you would have to manually implement expect/wait combo that asserts Animation has finished text has appeared. EarlGrey automatically delays the matching until a target view is in a stable state. This means that the check:

EarlGrey.select(elementWithMatcher: grey_text("Animation has finished!"))
      .assert(grey_sufficientlyVisible())

will fire as soon as (but not until) the transition and the animation are finished, regardless of how much time it takes. The framework automatically takes over when the animations are finished, so there is no time delay for sleep/polling.

Snapshot interaction testing

Up to this point, we have proven that the label is shown as a sign of animation being completed. But we still can't be sure whether the animated view looks exactly as expected. Is the rectangle visible? Does it have a correct shape? What is its background color?

Essentially, we are back to a problem that we have already solved with snapshots. Although EarlGrey has a built-in view snapshot feature, it is not something we can reuse easily. In order to do that, one would have to reimplement FBSnapshotTestCase's snapshot comparison logic.

Wouldn't it be amazing if we could somehow join FBSnapshotTest and EarlGrey together? I have got a great news for you! I have implemented a set of EarlGrey assertions which does that precisely. It is called EarlGreySnapshots.

Installation through CocoaPods is straightforward. Firstly, create a new target like you would usually do for your EarlGrey tests. Secondly, add following dependencies to that target:

target 'UITests' do
  inherit! :search_paths
  inhibit_all_warnings!

  pod 'EarlGrey', '~> 1.0'
  pod 'FBSnapshotTestCase', '~> 2.0'
  pod 'EarlGreySnapshots', '~> 0.0.2'
end

Now, execute pod install command and create an empty test file. For the sake of colors example, let's see how we can redefine our EarlGrey test to assert against a snapshot instead of a text:

import EarlGrey
import EarlGreySnapshots
import XCTest

class AnimatedRectangleFeatureTests: XCTestCase {
  func testRectangleViewIsAnimatedProperly() {
    EarlGrey.select(elementWithMatcher: grey_text("Animated Cyan"))
      .using(searchAction: grey_scrollInDirection(.down, 50.0),
             onElementWithMatcher: grey_kindOfClass(UITableView.self))
      .perform(grey_tap())
      .assert(grey_sufficientlyVisible())

    EarlGrey.select(elementWithMatcher: grey_kindOfClass(
                    AnimatedRectangleView.self))
      // replace with .assert(grey_recordSnapshot()) to capture reference
      .assert(grey_verifySnapshot()) 
  }
}

From now on, the code asserts if AnimatedRectangleView passes the snapshot test. Let's double check whether the reference image is truly produced in a stable post-animation state:

It is clear that the view state after the animation is asserted now.

Further considerations

EarlGrey together with EarlGreySnapshots can handle a plethora of scenarios that were not covered in this example post:

• User-interactions: taps, long presses, swipes, pinches and so on.
• Network calls (for integration testing).
• Delayed actions/operations triggered by NSOperationQueue or by using dispatchAfter.

It is a great toolset to enforce correct behavior of the app. As opposed to traditional tooling, it is much easier to use and way less cumbersome to write:

• You'll never have to worry again about asserting against multiple properties of numerous subviews in the same stable state.
• There is no need to assign dummy accessibility labels^[3] or setting the application up for UI testing in any special way.
• The tests does not rely on volatile expect conditions and are much harder to break.
• You can easily prove that your UI edge case handling hasn't broken.

I really recommend you to check it out now! 😉