Wicked Fast Testing: Part 1

Nov 24, 2020

“1 hour each day was spent rewriting the code. 7 hours was spent rewriting the tests.”

I’m really excited about this new series on Wicked Fast Testing. When I started CodeFaster, Wicked Fast Testing was one of the big ideas I wanted to share. It has saved me countless tedious hours of manually refactoring tests and changed the way I think about testing. Let’s jump into it.

Problem: Testing is slow

Wicked Fast Testing started because in 2017, I was shipping a new system. It was changing constantly. There were ever changing requirements. We discovered bugs in our vendor. We discovered bugs in the library that wrapped their API. We discovered undocumented features from our vendor. We discovered bugs in our code. And, often times, we just needed to refactor our codebase.

1 hour each day was spent fixing the code. 7 hours was spent fixing the tests. True, I had 200+ tests, but this system moved other people’s money. We couldn’t imagine accidentally taking someone’s money, so why should our system?

I was following industry standard techniques. My tests used the standard 3 As of Testing. The system was a Java based web service, so it had a lot of unit tests written in JUnit and a few http integration tests written in pytest. I had clever harnesses to help DRY up the tests. And it still took me 7 hours every day because I had hundreds of tests to work with. I’m all about coding fast, and testing was my bottleneck.

Wicked Fast Testing

cat expected.json | ./tester > actual.json && \
cp actual.json expected.json

It was in these dark times that Wicked Fast Testing was born. Wicked Fast Testing is a testing methodology. It will save you time no matter what programming language you’re using, what type of software you’re writing, and what type of test you’re writing (unit, integration, End-to-End, frontend, backend). In this post, we’ll just show an example of how to implement Wicked Fast Testing to test a simple method. In future parts, we’ll harden it to be more useful in practice.

A toy program

Imagine we have a method which prime factorizes a number. Here’s a simple implementation in python

# program.py

def prime_factorize(n, d=None):
    if d is None:
        d = {}
    for i in range(2, n//2):
        if n % i == 0:
            d[str(i)] = d.get(str(i), 0) + 1
            return prime_factorize(n//i, d)
    d[str(n)] = d.get(str(n), 0) + 1
    return d

Testing our toy

In Wicked Fast Testing, testing is centered around two files: expected.json and tester.

expected.json

[
  {
    "name": "prime_factorize",
    "parameters": [
      4849
    ],
    "result": {
      "13": 1,
      "373": 1
    }
  }
]

expected.json is our “json test file”. It contains all of the test data, both test parameters and expected results. This expected.json uses the following schema:

expected.json is a json array of “test objects”
Each “test object” contains the following keys:
- name - A string of the test name to run. These names are defined in tester
- parameters - an array of method of json serialized parameters for the test
- Result: A json serialized output of the test. Can be any json type.

In later parts of the series we’ll use more complicated schemas to handle more complex behavior, but in this part, we’ll keep it simple.

tester

#!/usr/bin/env python3

import json
import program
import sys

def prime_factorize(n):
    return program.prime_factorize(n)

def no_such_test(parameters):
    return "No such test found"

def test(test):
    test["result"] = globals().get(test["name"], "no_such_test")(*test["parameters"])
    return test

def t(data):
    return [test(d) for d in data]

print(json.dumps(t(json.load(sys.stdin))))

tester generates a “json test file” and populates each test’s result with the program’s actual output of the test. It’s used as follows:

cat expected.json | ./tester > actual.json

The tester is a json RLW (you can write it in your favorite language because every language supports json). It reads a json test file from STDIN and writes a json test file to STDOUT.

The fancy line above involving globals is a simple reflection trick to automatically map strings to methods defined in tester and default to “no_such_test” if a test isn’t found.

In later parts of the series we’ll add more functionality to tester.

Running the tests

First, use tester to generate a new json test file where result is the actual result returned by the program and store it in actual.json

# json-format pretty prints the json
cat expected.json | ./tester | json-format - > actual.json

expected.json contains the expected results and actual.json contains the actual results. This runs the first two As of testing.

To run the 3rd part of testing, assert, we need to compare the expected results to the actual results. If they’re the same, the test passes, if they’re different the test fails. Since expected and actual are json files, we can use json-diff:

json-diff expected.json actual.json

Each difference is a test failure. If there are no differences (json-diff returns an empty array), all tests passed. Alternatively, if we pretty print expected.json and actual.json with sorted keys (jq -S ‘.’), we can also use diff as follows:

diff expected.json actual.json

If there are differences, and we manually verified at least one is bad, we debug the code and generate actual again.

If there are differences, and we manually verified all of them are expected differences due to code changes, we can fix the tests with the following command:

cp actual.json expected.json

This is one reason why Wicked Fast Testing is so fast. The computer generates the results. Instead of filling them in manually, the computer does it. As humans, we just have to manually verify different outputs (or as we’ll see, write programs to verify them). The other reasons why Wicked Fast Testing is so fast will be uncovered in future installments.

Conclusion

That’s the basics of Wicked Fast Testing. In the upcoming posts, we’ll dig deeper. We’ll talk about how to write Wicked Fast Testing friendly tests. We’ll show how to do partial test runs, parallel tests, and exceptions. We’ll go into writing test verifiers to lighten the load of manual test verification. We’ll see examples of using Wicked Fast Testing for integration testing, front end testing, and http testing. If you don’t want to miss out on more Wicked Fast Testing, just hit the “Subscribe Now” button below.

CodeFaster