Fixing snapshot testing fatigue

This is part of a series on Wicked Fast Testing, a testing style that represents tests as JSON data. Check out Part 1 for a quick introduction.

Snapshot testing as used in the Jest testing framework and Wicked Fast Testing can really speed our coding, but has a curse: snapshot fatigue. In this post, we’ll dig into snapshot testing, snapshot testing fatigue, and how you can use Wicked Fast Testing, but not Jest, to solve this problem.

Snapshot Testing

Snapshot testing involves capturing the output, say of a UI component, and persisting it. Then future tests are tested against this persisted output. The real advantage of snapshot testing is that the snapshots can be updated by the computer. Imagine changing a small UI component and breaking 100 tests. Without snapshot tests, a human slowly and manually updates each test. With snapshot tests, a computer rapidly and automatically updates each test.

In the Jest testing framework, we can update the snapshots with:

jest -u

In Wicked Fast Testing, we can update snapshots with:

cp actual.json expected.json

Although snapshot testing is most common for testing UI components, the idea doesn’t have to only apply to UI components. We can use it for API results, DB state, and the return value from a function, assuming its serializable. Jest snapshots can be used for anything serializable into a string and Wicked Fast Testing can be used for anything serializable into JSON.

Snapshot Testing fatigue

Snapshot testing is great. Write a simple test that tests an entire output. Any changes can be instantly fixed with automatic snapshot updating. This sounds nice, until we have 100 tests and we change one meaningless thing about one method and now we have 101 failures to comb through. 101. 100 expected, meaningless changes and 1 unexpected, meaningful change that catches a bug. Want to find the one error? Start carefully looking through each failure. Sounds dull. And in practice, most fixes don’t have any errors. You sift through 100 changes to conclude everything was fine. As this happens more and more, you get tired of digging through so many changes. It’s probably fine you tell yourself. Who’ll notice? Well, one time, the customer did. It turns out, there was an unexpected change in test #73. You could have noticed, if only you went through each test painstakingly to find the needle in the haystack. However, going through tests painstakingly is slow and psychologically draining, so this isn’t a solution either.

Solution

At a high level, the solution to snapshot fatigue is simple, have the computer filter out the expected differences and only show the unexpected differences.

In Jest, implementing this is difficult. Jest snapshots are js files that export custom strings. This means the best existing tool to compare snapshots is diff. We diff (or git diff) between the old and new snapshot file. Then we have to filter out the diff entries that are expected which means processing diff. We might think filterdiff would help us out, but it’s only for filtering by diff filename, not filtering by diff content.

In Wicked Fast Testing, snapshots are json. This means we can use json-diff as follows:

json-diff expected.json actual.json

This outputs all of the differences, as nicely structured json. For example consider the following simple files

expected.json

[
  {
    "name": "f",
    "parameters": 1,
    "result": 1
  }
]

and actual.json

[
  {
    "name": "f",
    "parameters": 1,
    "result": 0
  }
]

then the output of

json-diff expected.json actual.json 

is:

[
  {
    "leftValue": 1,
    "path": [
      0,
      "result"
    ],
    "rightValue": 0
  }
]

This output is structured and easy to process. Both the left and right values easily accessible. The path is also structured as an array of strings or numbers which is also accessible. This means its easy to write a filter which filters out expected differences. To filter out all changes from 0 to 1, we could write the following jq:

jq 'map(select((.path[-1] == "result" and .leftValue == 1 and .rightValue == 0) | not))'

This is easy to combine with the output of json-diff:

json-diff expected.json actual.json | \
jq 'map(select((.path[-1] == "result" and .leftValue == 1 and .rightValue == 0) | not))'

This code is fast to write and straight forward to read. It filters out anything where “result” changed from 1 to 0. We can easily imagine for our hypothetical case of 101 bugs, including 100 identical changes, writing a similar filter to filter out the identical changes. The identical changes will be filtered out, but the bug will not be. It will show up clear as day, as if we burned our haystack and all that’s left is a glowing red needle.

Conclusion

In this post we discussed snapshot testing and why it’s so useful. We saw a major drawback where fatigue and dullness limits its usefulness. We were able to overcome fatigue in Wicked Fast Testing by using json-diff and writing custom filters. Because json-diff’s output is structured json, writing custom filters is very fast. We were not so lucky in Jest.

This concludes our series on Wicked Fast Testing. If you have any more questions about Wicked Fast Testing, message me on twitter @canardivore, I would love to help you.

Next week, we’ll dig into the json-toolkit. We’ve seen it show up quite often saving us valuable time. In this post, for example, json-diff was the lynchpin that solved snapshot testing fatigue. If you don’t want to miss out on learning how to use these time saving tools, just hit the Subscribe button.