Pinning tests in python using pytest

Don’t know what pinning tests are? Maybe you have heard about snapshot tests or characterization tests? They all refer to the same thing. Anyway, lets have a quick overview to fresh up on the idea.

Photo by Immo Wegmann on Unsplash

A pinning test is a very simple test that locks down (pins) the behavior of a specific execution path of a codebase, regardless of whether the output is actually correct or not.

A pinning test works in two phases:

• Initial run to pin down the current behavior (store it for later comparison).
• All subsequent runs compares the new output with the pinned output from the initial run. The test passes if the new output is identical to the pinned output and fails otherwise.

As already stated, a test like this won’t be able to tell you whether your code outputs is correct or not. So, what’s the point?

The point is that these tests are very good at guarding your codebase from unintentional and unanticipated changes! In my personal opinion, they come in very handy in the following situations:

• When refactoring, especially legacy codebases without any other test coverage.
• When dealing with code for numerical computations, such as optimization problems, simulations, classification, regression etc. where explicit unit and integration tests are notoriously hard to write except for the very simplest, analytically solvable, cases.

In both of these cases, the developer needs to be certain that any changes introduces to the codebase has or hasn't altered the output from the codebase according to his/her expectations and this is where pinning tests shine.

So, lets get to the point: How do we write pinning tests in python? And can we use our favorite testing framework, pytest, for this?

pytest-pinned to the rescue!

pytest-pinned is a pytest plugin and gives you the following:

• A single fixture to be used in your pytest test suite. pytest-pinned keeps track of in which test the fixture is used and allows for using the same fixture multiple times in a single test.
• A single json file collecting the pinned output from your pinning tests (this one should be added to your VCS!)
• pytest configuration flags for rewriting and updating the pinned output

All of which makes it very easy to get started.

On to an example. First, install pytest-pinned:

pip install pytest-pinned

I have kept this example very simple deliberately in order to focus on the functionality of pytest-pinned. We will be testing a function, my_func , that just returns a list of numbers. The test function makes use of the pinned fixture from pytest-pinned and compares the result from my_func with the picture using the is-equal-operator == .

Make an initial run to pin down the behavior of my_func :

pytest --pinned-rewrite test_my_func.py

yields this json file:

Make sure to keep this file version controlled, otherwise pinning tests kind of lose their charm.

To run the tests and actually do the comparison against the pinned behavior (this is what should be in your CI pipeline!), run the following

pytest test_my_func.py

For more details, check out the pytest-pinned repo. For inspiration on how pinning tests can be used when dealing with numerical simulations, have a look here and here.