Generating tests (semi) automatically

This is part of a series on Wicked Fast Testing, a testing style that represents tests as JSON data. Check out Part 1 for a quick introduction.

Generating tests semi-automatically is important because manually typing out test cases is a drag on coding speed, even with vim. Unfortunately generating useful tests fully automatically from code is a significantly difficult problem.

In this post, we’ll show how with a few lines of python, we can generate dozens or hundreds of tests.

What is test generation?

Test generation is using a computer to write tests. It’s not enough to generate the raw data or the test case, but the entire test. Since Wicked Fast Testing supports snapshot testing, we already have a solution to generating the expected test values. Therefore the remaining open problem is generating the test case as well as formatting into a test. Formatting a test case into a test is easy with Wicked Fast Testing because its tests are a json array of well structured json objects.

Why generate tests?

Writing tests by hand is slow, especially at scale. It’s not the typing that’s slow, if a test takes 10s to write, 100 tests takes 20 minutes which is not so bad (although a computer, once programmed, can write millions of tests in 20 minutes). The real slow part is the thinking, and the errors.

Generating tests by hand requires remembering which test we’re at and which one is next? “Well last one was [-1, 0, 5] so the next one should be… [-1, 5, 4]? Wait no… I did that one…” Doing this fast requires a strong working memory and being very organized in thought all the time. And when you do it right, you’re still slower than a computer

Generating tests by hand is also error prone. Human brains can move faster, but then we’re more likely to make mistakes. We can make errors at a number of levels. Which test should I write, which character within the test should I write, fat fingering the wrong character, skipping a character, etc. So either we move slowly and carefully or quickly but recklessly. Humans at their best, using either technique, will be slower than a computer which types fast and accurately.

How

Cross Product technique

The first type of generation is “cross product”.

Let’s say we have a function, f, that takes in two parameters, a, and b.

f(a,b)

We want to test when a is negative, 0, and positive, as well as when b is 0, odd, and positive even, then we have 3 test values for the first parameter, and 3 for the second. To test all combinations is to test 9 values, we need the cross product of [-1,0,1], for a, and [0, 1, 2], for b.

Generating cross products is simple, just use nested for loops:

#!/usr/bin/env python3

import json

def t():
    o = []
    for i in [-1, 0, 1]:
        for j in [0, 1, 2]:
            o.append([i,j])
    return o

print(json.dumps(t())

This outputs

[[-1, 0], [-1, 1], [-1, 2], [0, 0], [0, 1], [0, 2], [1, 0], [1, 1], [1, 2]]

To fully generate the test, we just modify the argument to o.append:

#!/usr/bin/env python3

import json

def t():
    o = []
    for i in [-1, 0, 1]:
        for j in [0, 1, 2]:
            o.append({
                "name": "f",
                "parameters": [i,j],
                "run": True,
                "result": None
            })
    return o

print(json.dumps(t()))

and we invoke it with

./generate-cross-product | json-format - > tests.json

Template copy

This generation technique involves writing a simple test case by hand, and then using a computer to generate more tests from it. Let’s consider a simple test for f which is stored in test.json

{
  "name": "f",
  "parameters": [
    0,
    0
  ],
  "run": true,
  "result": null
}

We want to make more tests from it. Let’s say we want the first parameter to vary from -10 to 10. We can use the following simple program, template-copy:

#!/usr/bin/env python3

import copy
import json
import sys

def t(data):
    o = []
    for i in range(-10, 11):
        test = copy.deepcopy(data)
        test["parameters"][0] = i
        o.append(test)
    return o

print(json.dumps(t(json.load(sys.stdin))))

which we invoke with

cat test.json | ./template-copy | json-format - > tests.json

21 tests generated fast.

Combining test files

The above techniques work great, but each program that generates tests outputs its own test file. We want one test file, so how do we combine test files?

Combining files is a tinge more complicated than using cat because, unlike DSV or CSV, json arrays are not newline delimited records. However, jq can help us out here.

cat tests_1.json tests_2.json \
| jq 'flatten(1)' \
| json-format - \
> combined-tests.json

Assuming tests is an array of objects, It’s not important to use flatten(1) over flatten. But if tests is an array of arrays, flatten does not only combine the tests but it will also flatten the arrays within each test which is not desirable.

Conclusion

We’ve just shown how we can use a few lines of python to generate dozens or even hundreds of tests. We’ve shown two techniques: cross product, which allows us to test every combination of a few variables, and template copy, which allows us to modify a single test to create many tests. We can even combine these techniques. Finally, we showed how to combine test files into a single file. This is important because test generation

Next week, we’ll show how to programatically solve snapshot testing fatigue, a serious concern for any user of Jest or Wicked Fast Testing, although it can only be solved in Wicked Fast Testing. If you don’t want to miss out on becoming a faster coder, just hit the subscribe button below.