Testing

Mantid Imaging uses unit tests, static analysis and GUI approval testing

The full test suite can be run using:

make check

Tests are run automatically on pull requests on GitHub using actions, but should also be run during development.

Pre-commit

Mantid Imaging uses pre-commit to run pre-commit hooks. This is included in the development conda environment and can be installed by using pip install pre-commit. After installation, you can then install the hook scripts with pre-commit install. Once this has been completed, the hooks should execute with every commit.

Unit testing

Unit tests can be run using pytest, e.g.

python -m pytest

For options such as running a subset of tests, see PyTest Docs

Static analysis

Mantid Imaging uses mypy, flake8 and yapf for static analysis and formatting. They are run by make check, or can be run individually, e.g. make mypy.

GUI screenshot testing

Mantid Imaging uses Applitools Eyes for GUI approval testing. Screenshots of windows are uploaded and compared to known good baseline images. This is run in the github action on pull requests.

Appplitools requires an API key to use, which can be found on via a Applitools web interface. On a developer machine this can be passed as an environment variable. E.g.

APPLITOOLS_API_KEY=XXXXXXXXXX xvfb-run --auto-servernum pytest -p no:xdist -p no:randomly -p no:repeat -p no:cov mantidimaging/eyes_tests

Differences between uploaded and baseline images can be examined and approved or rejected from the Applitools web interface.

To run without a key or to prevent uploads, set APPLITOOLS_API_KEY to local and choose a directory to save the screenshots. Note that this does not check for changes, and will always pass. e.g.

mkdir /tmp/gui_test
APPLITOOLS_API_KEY=local APPLITOOLS_IMAGE_DIR=/tmp/gui_test xvfb-run --auto-servernum pytest -p no:xdist -p no:randomly -p no:repeat -p no:cov mantidimaging/eyes_tests

GUI system tests

GUI system tests run work flows in Mantid Imaging in a ‘realistic’ way, where possible by using QTest methods to emulate mouse and keyboard actions. They use the same data files as the GUI screenshot tests. These take several minutes to run and so must be explicitly requested.

pytest -v --run-system-tests

or in virtual X server xvfb-run

xvfb-run --auto-servernum pytest -v --run-system-tests