Development tools

MUSCLE3 is a fairly complex piece of software, being a coupling library designed to link together independent bits of software written in different languages. To help us develop MUSCLE3, we use a variety of tools that check our work and build software and documentation where required. This page describes the development tools we use, what they do, and how they are configured.

Building and installation

The main language for MUSCLE3 is Python 3. To help install and publish the Python parts of MUSCLE3, we use setuptools. Setuptools reads information from the file setup.py in the root of the repository. This file contains a description of MUSCLE3 which is used on PyPI (the standard online Python package repository), information used by PIP when installing MUSCLE3, and information about required libraries.

The dependencies in setup.py are specified with an exact version number, rather than an at-least version number. This way, we can be sure that no incompatible changes are introduced that break MUSCLE3. However, it also means that the dependencies get out of date, as new versions are released. To keep up-to-date on this, we use Requires.io, a web service that scans our setup.py and checks our versions against the latest releases. It produces a button, which we embed in our README.rst, that shows whether we are up-to-date.

We will most likely have outdated dependencies quite often, as rapid updates are a fact of life in today’s Open Source world. This is not an immediate problem, as the older versions will still be available and MUSCLE3 will work with them, but we should update regularly to stay current and avoid potential security issues.

Quality Control

MUSCLE3 is infrastructure used for building infrastructure (models) for doing scientific experiments. As such, it is important that it is reliable, easy to use, and easy to develop. Furthermore, it will frequently be used by inexperienced programmers, who will look at MUSCLE3 as an example of how to write good code. It is therefore important that MUSCLE3 is a well-implemented, high quality program. To make sure that this happens, we use static checking (linting), unit tests, and integration tests.

Static checking

The quality control system uses several tools, which are called from the central Makefile. In our standard development environment, the command make test in the top directory will run them all. This in turn runs tox, which runs the individual tools.

Python has a standard coding standard known as PEP8, which specifies how Python code should be formatted. Consistently writing our code according to PEP8 makes it easier to read, and makes it easier for new developers to get started. We use ruff to check the formatting.

Although Python is a dynamically-typed language, it supports type annotations, and using those for static type checking. While writing code with type annotations sometimes takes a bit more thinking than doing without them, it increases the reliability of the program a lot, as type checking can detect mistakes that unit tests don’t. So we use type annotations, and use mypy to check them as part of the set of quality control software.

Finally, we use Codacy, which is an online tool that runs various static checking tools against our repository. It is run automatically for the main branches, as well as for every pull request. Codacy tends to give false positives fairly often, but the settings can be tuned and specific warnings can be set to be ignored. So having some Codacy warnings on a pull request is not necessarily a problem, but they should be reviewed and either fixed or disabled as appropriate.

Tests

MUSCLE3 has a suite of tests, both unit tests (which test small bits of the code one piece at a time) and integration tests (which test whether it all goes together correctly). We run these tests, measure test coverage, and have continuous integration just in case we forget to run the tests manually.

The main test framework for the Python part of the code is PyTest. This Python library provides a library with functions that make it easier to write test cases, and a runner that automatically detects tests and runs them. Tests are located in test/ subdirectories in the code, and in the integration_test directory in the root of the repository. In various places in the code, files named conftest.py are located, which contain test fixtures. These files are picked up automatically by PyTest.

We use a code coverage plug-in for PyTest, which measures which lines of the code are executed during testing, and more importantly, which lines aren’t. Ideally, all code is covered, but in practice there will be things that are difficult to test and not important enough to warrant testing. 90% coverage is a nice (but sometimes challenging) target to aim for. This plug-in uses the Python coverage library, which is configured in .coveragerc, where e.g. generated code is excluded from testing.

Tests can be run locally using make test, but are also run in the cloud via Github Actions on every push to the Github repository. After Github has run the tests, it reports the code coverage to Codacy, which integrates it with the linting results into its dashboard. It also reports back to GitHub, adding an icon to e.g. a pull request to signal whether the tests have passed or not.

There’s a special set of tests for testing MUSCLE3’s HPC cluster support. These require Docker to be installed, they’re run on Linux only, and are typically run only just before a new release is made, because they are very expensive. On the first run, Docker containers get built, which make take several hours and a few GB of RAM, and the tests themselves take up to half an hour to complete.

To run these test, use make test_cluster. If any of them fail, edit integration_test/cluster_test/conftest.py and set CLEAN_UP_CONTAINERS to False and run again. You’ll probably also want to reduce the list of SLURM versions in IDX_SLURM_VERSIONS down to a single one, to speed up the debugging process.

Once the tests have completed, you’ll have a set of docker containers still running. Use ssh to connect to a headnode container with username cerulean and password kingfisher, e.g. ssh -p 10022 cerulean@localhost. You’ll find the test results under /home/cerulean/shared/test_results; the logs in there should show what went wrong so that you can fix the issue.

Note that a successful run will still result in an Operation not permitted error during clean-up, because the container is still running and therefore the shared folder cannot be removed. Setting CLEAN_UP_CONTAINERS back to True will fix that.

Documentation

The MUSCLE3 documentation lives in the repository together with the code. This is a much preferable arrangement to a separate wiki, as previous MUSCLEs had, because wikis tend to get lost.

The source for the documentation is in docs/source, and it is written in ReStructuredText. It is processed into HTML by the Sphinx program, which uses the sphinx-autodoc plug-in for extracting docstrings from the Python source code and converting that into API documentation. Sphinx is configured via docs/source/conf.py. This file contains a hook to automatically run sphinx-autodoc whenever Sphinx is run, so that is does not have to be run separately first.

The ReadTheDocs online service is configured to automatically render the documentation and display it online. This uses Sphinx, but does not support sphinx-autodoc, which is why we use the aforementioned hook.