“Snapshot testing” is a phrase that you’ll hear in circles of developers who are into unit testing. Conceptually, snapshot testing is a simple idea: take a screenshot of your user interface and store it as a reference image. Then, make unit tests to take screenshots of the same UI and compare it to the reference image. Facebook created, and Uber now maintains, ios-snapshot-test-case, a testing framework that makes this process really easy. Lots of us have written testing tools on top of ios-snapshot-test-case and have been using it in production for a few years now.
So here’s the question that I want to discuss: is it worth it?
tl;dr yeah it is totally worth it, but (like any technique) you want to be careful about over-doing it.
The idea to write this post came from a friend who shared this trick to display before and after screenshots in GitHub pull requests using Markdown tables.
🔥 Pull Request Description Tip— Gio 💻🔧📚 (@mokagio) July 18, 2018
Using tables is useful for showing before/after screenshots in PRs touching the UI. pic.twitter.com/lYhfQX30T5
The conversation turned to snapshot testing; a consequence of snapshot testing would be that you don’t need to include before/after shots in your PR description because they’d be included for you in the PR’s changed files. Think about it: your reference images are essentially screenshots of your UI – and they’re stored in your repository. So when you open a PR that changes the UI, your PR will necessarily include changes to those references images, showing the before and after of the UI changes. Neat!
That’s awesome!— Gio 💻🔧📚 (@mokagio) July 19, 2018
I was considering a spike to set up some snapshot testing at work too.
Would you say it’s worh the time?
Does it often reveal defects like that? False positive?
Is it worth the time? Yes, for our team it was.
Does it reveal defects? Yes it does – let me give you an example from last week at Artsy.
We have this button class,
ARBidButton, which is given an
ARAuctionState and is responsible for updating its background colours, text colours, enabled state, and text. We include this button on artwork views in auctions so the user can bid on them. We wanted to change some of the render logic in the button, but we inadvertently broke another part of our app that we didn’t realize was using
ARBidButton too. Here is the original pull request and here is the second attempt we made (once we realized the button was used elsewhere).
If we hadn’t had snapshot tests for different parts of our UI, we wouldn’t have noticed that the small change we wanted to make would have inadvertent changes elsewhere in our app.
But what about drawbacks? Is snapshot testing worth it? Well, let’s back up and get more context.
Developers who are familiar with different testing strategies eventually form an intuitive understanding of what’s formally called the “testing pyramid.” Here is a depiction of the pyramid that I borrowed from this unit testing book.
The idea is that some tests are really quick to run (automated unit tests) and others take a really long time to run (manual tests). The testing pyramid models the spectrum between the two: if a test is expensive to run, the pyramid tells you to run fewer of those tests. The pyramid helps developers balance the competing demands of testing (to ensure quality) and shipping quickly (to stay in business).
Snapshot testing sits right on that barrier between “did we build it right?” and “did we build the right thing?” That means that, while snapshot testing is useful, it shouldn’t be your entire testing strategy. You will still need unit tests and manual tests, but snapshot tests really help fill in the gap between the two.
So, with the testing pyramid in mind, let’s return to the question: is it worth it? Well, let’s run down some quick pros and cons.
- Pro: Reference image changes provide an opportunity for designers to give feedback on pull requests.
- Con: Reference images have a large filesize and can change frequently, which can lead to a ballooning git repository size (our repo that makes the heaviest use of snapshot testing is 500MB on disk).
- Pro: Lots of tools exist to make snapshot testing even more useful.
- Con: Tooling for snapshot testing, such as this project, are hampered by Xcode’s extremely limited plugin API.
- Pro: Snapshot tests are a very broad-strokes confirmation that your UI is “correct.”
- Con: Snapshot testing can only test static UIs and not things like animations or user interaction.
- Pro: Testing across devices, iOS versions, and size classes becomes trivial.
- Con: Snapshot testing introduces additional complexity to your testing setup.
- Pro: iOS-snapshot-test-case has tools for allowing a specified percentage of differing pixels, along with other customization options.
- Con: iOS versions often include changes to text antialiasing, so your tests need to be tied to specific iOS versions. Updating reference images when you update your iOS versions introduces a lot of churn in reference image files, contributing to repo size bloat.
- Pro: Snapshot testing encourages you to write testable view controllers.
- Con: It’s easy to overdo snapshot tests, feel confident in your app’s correctness, but not actually be testing the correct thing.
- Con: Snapshot test failures on CI can’t be reviewed, since the image files aren’t accessible (it’s not difficult to get around this with an S3 bucket).
I have no regrets about our snapshot testing strategy, but others will have different approaches. A lot of the cons that I’ve outlined above have workarounds (one could, for example, store reference images in a submodule to mitigate repo bloat). And snapshot testing is really an open field: if you have an idea for a tool, you could probably build it on top of tools that already exist.
If you decide to try snapshot testing, I can’t recommend Kaleidoscope enough. Its integration into
git difftool makes reviewing changes to reference images before opening a PR really easy.
I hope that gives you a clear picture of what snapshot tests are and how they can be used as a part of a comprehensive testing strategy. It’s a cliché, but snapshot tests are not a silver bullet. They will not be a replacement for unit testing. But they can help you write better code, be more confident in your product, and open up whole new ways to collaborate with non-developer colleagues. They’re definitely worth learning about – a tool worth having in your toolbox.