Skipping tests

Dec 01, 2020

This is part of a series on Wicked Fast Testing, a testing style that uses the computer to refactor tests saving you countless hours. Check out Part 1 for a quick introduction if you’re not familiar.

Skipping tests is important because waiting for unnecessary tests to finish is a drag on coding speed. However, since this is Wicked Fast Testing, we’ll even make it wicked fast to specify which tests to skip.

In this post, we’ll show how in 1 line of jq we can perform simple test skipping. With 2 lines of python and 1 line of jq, we can implement test skipping with automatic test updating.

Why skip tests?

The fundamental metric for coding velocity is coding feedback loop cycle time. This deserves a post on its own, so for now we’ll take it on faith.

The “development feedback loop cycle time” is the amount of time it takes to code and get feedback on it. The most common case of this is the code-test cycle time where we get feedback from our tests if we wrote any bugs. This cycle time is composed of two simple times: code time and test time.

Skipping tests, especially tests that drag their feet and don’t provide feedback, can decrease the test time. Of course, if we err and skip tests that could provide feedback, it means we’ll get the feedback later. Perhaps from a customer in production.

In practice with traditional testing frameworks, there’s one big problem: specifying which tests to skip is heavy, manual labor. Heavy, manual labor retards our otherwise fast “code-test” cycle. In a traditional testing framework, here are the best ways to express which tests should skip, or equivalently which other tests to run:

Run only tests in this file. This lets us skip entire files, but not part of a file
Run only failed tests. This lets us skip tests which passed in the past.
Run only tests marked with a decorator. This lets us specify an arbitrary list of tests, but comes with the overhead of adding and managing the decorators manually.
Run a specified list of tests. This lets us specify an arbitrary list of tests, but comes with the overhead of adding and managing the test names manually.

As we see we have a tradeoff between fast to specify, coarse blocks (run all, run a file, run failed) or fine grained blocks with a large typing burden. Typing out many names by hand or copy/pasting a decorator feels like programming with a hammer and chisel. It gets the job done. Eventually.

But we’re programmers with computers, we can do better. Much better.

Wicked Fast Testing: Skip tests

With Wicked Fast Testing, test skipping is easy. Let’s say we’re testing two methods, foo and bar. expected.json, our test case file will look like

[
  {
    "name": "foo",
    "parameters": [ 0, 1, 2 ],
    "result": true
  },
  {
    "name": "foo",
    "parameters": [ 0, 1 ],
    "result": false
  },
  {
    "name": "bar",
    "parameters": [ 0, 1, 2 ],
    "result": true
  },
  {
    "name": "bar",
    "parameters": [ 0, 1 ],
    "result": true
  },
  {
    "name": "foo",
    "parameters": [ 0, 2 ],
    "result": true
  }
]

and as a quick reminder we run all tests with

cat expected.json | ./tester | json-format - > actual.json

json-diff expected.json actual.json

If we want to run only tests which test foo, it takes one line of jq.

cat expected.json | jq 'map(select(.name == "foo"))' | json-format - > foo-expected.json

cat foo-expected.json | ./tester | json-format - > foo-actual.json

json-diff foo-expected.json foo-actual.json

To specify which tests to run, we wrote a simple, short program. The program is fast to code, runs fast, and requires no manual management test annotations.

We can use a program to specify an arbitrary selection of tests based on anything in the test. If we wanted to run only foo tests where the first parameter was 0, it’s just as fast.

cat expected.json | jq 'map(select(.name == "foo" and .parameters[0] == 0))' | json-format - > foo-expected.json

Using programs to specify which tests to run isn’t like hammer and a chisel programming. It’s jackhammer programming. It can crunch through thousands of tests in less time than it takes you to add one test annotation manually. But there’s one problem. One of the cool features of Wicked Fast Testing is that when all of the differences between the actual output and expected output are due to the expected outputs being out of date, we can update them automatically:

cp actual.json expected.json

With large test suites and code churn, there can be hundreds of out of date lines. Fixing them by hand takes hours. cp runs in under a second. This saves us plenty of time and tedious work, so it’d be a shame that we’d have to choose between it and waiting for unnecessary tests. To fix this, we’ll need 2 lines of python, and 3 lines of jq.

Skip tests with one step stale test fixing

To implement test skipping while preserving automatic test updates, we’re going to modify tester with 2 lines of python. Here’s an tester to test foo and bar.

#!/usr/bin/env python3

import json
import program
import sys

def foo(args):
    return program.foo(*args)

def bar(args):
    return program.bar(*args)

def no_such_test(parameters):
    return "No such test found"

def test(test):
    test["result"] = globals().get(test["name"], "no_such_test")(*test["parameters"])
    return test

def t(data):
    return [test(d) for d in data]

print(json.dumps(t(json.load(sys.stdin))))

We’ll add our two python lines at the beginning of test.

def test(test):
    if not test["run"]:
        return test

    test["result"] = globals().get(test["name"], "no_such_test")(*test["parameters"])
    return test

These two lines mean any test marked with run false will have their actual result match their expected result. Therefore if we copy the actual file to expected, unrun tests will not change! However, we have a new smaller problem: this new run parameter. It’s not in our expected.json file. We could solve this one of two ways.

We could assume no run field for a test means run the test, just like how the lack of a skip decorator in a traditional test means don’t skip. This means our tests work as is, but any code dealing with them is more complicated because it can’t assume the skip field is present.
We can add the run parameter to every test. This is like adding a don’t skip decorator to every test in a traditional test. In a traditional tests, adding decorators like this can turn into a hammer and chisel problem where we’re manually adding a decorator to each test. In theory, if the code is clean and we’re handy with regular expression and sed, we can use them to automate a large part of the work. Even then, we might need to hammer and chisel away false positives and add in false negatives if the regular expression isn’t perfect.

We’re not dealing with a traditional test framework, we’re using Wicked Fast Testing, so we’ll pick option 2. It’ll take us 1 line of jq.

# jf is my personal bash alias for json-format

cat expected.json | jq 'map(.run = false)' | jf - > tmp
mv tmp expected.json

No fancy regular expressions or cryptic sed commands, just a straightforward jq command.

Let’s go back to our earlier example of running only the foo tests. With this new run field, we can run only the foo tests by with our last two lines of jq

cat expected.json \
| jq 'map(.run = (.name == "foo"))' \
| ./tester \
| jq 'map(.run = true)' \
| jf - > actual.json

json-diff expected.json actual.json

And now we’ve successfully performed test skipping while preserving the option to automatically update our tests. In practice, I wrap all of this into bash script run-tests and edit it as needed.

What we’ve done is, honestly, pretty mindblowing. Testing is now just running programs. Nothing manual. Picking tests isn’t manual. Fixing stale tests isn’t manual. No matter how many tests we have, the same simple short programs work. We’ll spend almost no time managing tests or ever have to mindlessly fix a hundred out of date tests. We’ll just spend our time coding, a few seconds running the tests, and update them all with a single swoop.

Why 2 jq commands

The second jq command setting run to be true after a test run serves a subtle, but important purpose.

Without it, copying actual.json to expected.json will change unrun tests’ run parameters to false. However, in a checked in expected.json, all of its run fields should be true.

If a test shouldn’t be run at all because it’s broken for reasons that can’t be addressed right now, but long term it should be fixed, delete the test (in a standalone commit or standalone pull request). Don’t leave it around in the database with run = false. To track that the test is broken and should be fixed later:

Create a ticket in a work tracker (like Asana)
Include the git commit hash of the standalone commit that deleted the test (and only deleted the test).
When someone is working on the ticket, they can git revert $COMMIT to add the test back and then begin working on the fix. If there’s a conflict, extracting the test from the commit is relatively straight forward using the following script:
```
git show $COMMIT | grep "^- " | cut -d '-' -f 2-
```
This pipeline works as follows:
- git show $COMMIT will show the test lines with “-” at the beginning of the line to show the lines were deleted.
- grep “^- ” selects only the test lines that started with “-” and removes any git show overhead. This is why it’s important to remove it in a standalone commit, otherwise all other removed lines from the commit will pass this stage of the pipeline.
- cut -d '-' -f 2- selects everything after the first -, which is the test (text) itself

Conclusion

We have just shown how to implement test skipping in Wicked Fast Testing in a way that preserves automatic test updating. Next week, we’ll discuss automatic test generation. If you don’t want to miss out on becoming faster through Wicked Fast Testing, just hit Subscribe Now below.

CodeFaster

Discussion about this post