Writing Tests#

Unit tests are an important part of writing software. Their usefulness comes in many ways:

it allows you to make quick verification of the changes applied
it makes sure that the code you are using is correct
it speeds up refactoring and optimization of the code

Therefore it is important to have many tests, which helps identifying potential issues. Below we present some technical aspects of testing. The list is by no means complete and if you’re interested, please check the pytest documentation.

General rules#

There are several simple rules that should be followed in order for unit testing to serve its purpose:

Unit tests should test single functionality in a meaningful way,
All public methods should have their own tests,
All lines should be tested by at least one test (code coverage), and
Tests should be as fast as possible (yet meaningful)

The last point is particularly important as all tests are performed on the continuous integration (CI) service, on which we have large, yet limited number of computational resources available.

Furthermore, having too many slow tests prohibits effective refactoring and optimization. On the other hand, speed is not the main factor: if you see that increasing time is essential for making good tests, feel free to write such a test (but mark it as slow as explained below). In any case, a single test should not take more than a few seconds. Ideally one test case should run in a fraction of a second.

Technical aspects#

Naming convention#

All tests should be placed in the tests folder. pytest test discovery analyzes the whole repository looking for Python scripts whose names are of the form test_<whatever>.py.

Warning

You should always run the test scripts as:

pytest test_<whatever>.py

and never as:

python test_<whatever>.py

Running in “isolation” will, in many cases, not work as you’d expect.

All test functions within the files should have names following the pattern test_<whatever>. Optionally, tests can be grouped inside classes whose name follows the pattern Test<whatever>.

Simple tests#

Within a test function, checks are performed using the assert statement:

def test_addition():
    assert 2*2 == 4

The test passes if no error occurred when it was run and all assertions are satisfied. If error was the expected behavior, you can use pytest.raises:

import pytest

def test_value():
    with pytest.raises(ValueError, match=r'must be \d+$'):
        raise ValueError("value must be 42")

The first argument to pytest.raises is the exception type. While it can be omitted, it is bad practice to do so,¹ as the test will pass for any other exception that might be raised by the code you are trying to test.

The second argument, match, is useful to further narrow down the exception we are trying to test. You should pass is a string or a regular expression (like in the example) to match against the error message reported by the exception. When trying to test raised standard library exceptions (like ValueError or TypeError) it is good practice to also use the match argument to pytest.raises.

The `pytest` command line#

You can alter the default behavior of the pytest invocation with command line arguments. Some useful options:

-x run pytest until first error encountered
-m "not slow" only run tests not marked as slow.
-m "slow" only run tests marked as slow.
--reruns=[m] rerun failed tests at most m times (useful if there are flaky tests)
--durations=[m] show m slowest test durations
--durations-min=[m] minimal duration in seconds for inclusion in slowest list, defaults to 0.005
-v run in verbose mode. You can add multiple vs to increase the verbosity further
-s to not capture stdout. Useful if you are using print for debugging

Marking tests#

We may organize tests in our test suite more effectively by adding marks to them. Below we present a list of marks which may be particularly useful. All the marks should be placed right before the method.

@pytest.mark.slow: indicates that the test function will take a long time to execute.
@pytest.mark.skip("[Message]"): skip the test. In place of [Message] you should write the reason why the test is skipped. Please use this mark only if absolutely necessary.

@pytest.mark.parametrize("[varname]", [val1, val2, ...]): parametrize the test over different values of the arguments to the test function. See a simple example below.

import pytest

@pytest.mark.parametrize(
    ("test_input", "expected"),
    [
        ("3+5", 8),
        ("2+4", 6),
        ("6*9", 42),
    ],
)
def test_eval(test_input, expected):
    assert eval(test_input) == expected

Collecting test cases with `pytest-cases`#

Test parametrization is a powerful technique to avoid code repetition when writing tests. However, it can be hard to understand exactly what and how it’s being parameterized. pytest-cases (see documentation) allows to collect the different parametrizations into case functions/classes, which can the subsequently be (re-)used in a parameterization. This leads to tidier test scripts.

See this example from the pytest-cases documentation:

from pytest_cases import parametrize_with_cases, case

class Foo:
    @case(tags=["one"], id="1")
    def case_a_positive_int(self):
        return 1

    @case(tags=["two"], id="2")
    def case_another_positive_int(self):
        return 2

@parametrize_with_cases("a", cases=Foo)
def test_foo(a):
    assert a > 0

In the parameterization, test cases can be filtered using tags or by globbing function names.

Info

By default, pytest-cases only considers function names following the pattern case_<whatever> to be valid test cases.

Reusable testing functionality#

Reusing data with test fixtures#

Often times you find yourself writing tests that use the same input data to perform their checks. In order to reduce code duplication, the standard approach is to use test fixtures. It is very easy to create fixtures with pytest:

import pytest


@pytest.fixture
def first_entry():
    return "a"

and using them is a matter of adding an input argument to your test function:

def test_string(first_entry):
    # Assert
    assert first_entry == "a"

Fixtures can be defined in the same file as the test or placed in conftest.py files in the same or any parent folder of the test file. conftest.py files are special files for pytest: they are “evaluated” during test collection, such that any globally defined data is available to the test functions when they are executed.²

Fixtures can be used as arguments to other fixtures. Also, they can be scoped, to decide when the fixture should be executed. By default, each fixture is function-scoped and is executed before every test function that needs it. However, it’s better to execute expensive fixtures fewer times and cache their result. This is achieved using the scope keyword argument to the the pytest.fixture decorator. The available scopes are documented here.

Reusing functions with `pytest-helpers-namespace`#

Other times what you want to reuse is not some data produced by a function, but the function itself. A typical use case is as follows:

You’re creating a parameterized test.
The value of the parameter is computed through other parameters.

pytest offers functionality for this in terms of parameterized fixtures and indirect parameterization, however:

The former requires one central place (the fixture definition) where to list all possible combinations of valid parameters. When running the test using the parameterized fixture, all parameter combinations will be executed. Excluding cases to run is cumbersome.
The latter results in a rather unintuitive way of preparing data for a test. Especially for those use cases where multiple arguments are needed to prepare the data for the test.

While parameterized fixture and indirect parameterization are very useful, pytest-helpers-namespace offers a lightweight alternative. One can define functions in any of the conftest.py files and decorate them appropriately:

import pytest


@pytest.helpers.register
def foo(bar):
    return bar

such that they can be reused in the test files, without having to import conftest.py first:

def test_helper_namespace():
    assert pytest.helpers.foo(True) is True

How to use logging with tests#

Within aurora, we adopt a pytest configuration that allows to see the output from the logger when executing tests:

[tool.pytest.ini_options]
log_cli = true
log_cli_format = "%(asctime)s %(levelname)s %(message)s"
log_date_format = "%Y-%m-%d %H:%M:%S"

The default logging level is logging.WARNING.

When writing your tests, you might find useful to obtain output to screen to check what is going on with the code you are trying to test. Traditionally, one would reach for the print function, but we explicitly discourage such uses and favour the use of logging instead. Also, while we don’t require it, you might want to test the log messages emitted by your code, which would require capturing the output from the logger within the test function.

The caplog fixture from pytest comes to the rescue in both cases. The following example shows how to set the log level for the aurora.chemistry.eos submodule to logging.INFO:

def test_foo(caplog):
    caplog.set_level(logging.INFO, logger="aurora.chemistry.eos")

Running the test on the command line will show logging output from the submodule. The fixture has attributes records and record_tuples, which store the data sent to the logger. These can be used if you want to check that specific log messages have been emitted by your code.

luckily, Ruff will flag this bad practice. ↩
this is why you should never think of the test scripts as usual Python scripts! Details of how conftest.py works can be found here. As a corollary to this, never add the boilerplate:
```
if __name__ == "__main__":
```
to the bottom of the test file. You can execute any test through pytest by simply selecting it with the -k flag. ↩