Skip to content

Files

benchmarks

MDAnalysis benchmarks

Benchmarking MDAnalysis with Airspeed Velocity.

Usage with MDAnalysis

Airspeed Velocity builds clean conda environments to benchmark the performance of MDAnalysis at different time points in its history (for different git commit hashes).

At the time of writing, Airspeed Velocity expects the project setup.py to be in the root directory of the git repository. MDAnalysis has had setup.py for the main package in a subdirectory for quite some time now, so for the time being we are using a modified branch (flexible_build) of Airspeed Velocity in our fork of their package, which enables Airspeed Velocity to build MDAnalysis despite the location of setup.py. A WIP PR is open to enable more flexible Airspeed Velocity build mechanics, but for now we will use our fork.

To build / install our fork of Airspeed Velocity it should suffice to clone the git repo, checkout the flexible_build branch, and run:

python setup.py install --user

Airspeed Velocity commands are described in detail in their documentation. A common usage example for evaluating the performance of a feature branch / pull request would be:

asv continuous --bench GROReadBench d76c9348 e0bc303 -e

In the above, GROReadBench is handled as a regular expression for the specific benchmark test to run between the provided git commit hashes.

To evaluate a benchmark test over the course of project history one would commonly use asv run. For example, to probe performance for trajectory readers at 20 commit hashes evenly spread over a few years of the project one might run:

asv run -e -s 20 ddb57592..e0bc3034 --bench TrajReaderBench -j

It is also possible to specify ALL to space the performance tests over the entire lifetime of the project, but exercise caution as very early commits may represent a state of the project where many features are not available and / or files are not in the expected locations.

The asv run command will store detailed benchmark data locally as JSON files, which can be converted into interactive website data and hosted locally with:

asv publish
asv preview

Writing benchmarks

The Airspeed Velocity documentation for writing benchmarks is a suitable reference. As a quick summary of guidelines:

  • wrap imports from MDAnalysis in the test modules because older commit hashes may not have the name imported for various reasons:

    try:
       from MDAnalysis.coordinates.DCD import DCDReader
    except ImportError:
       pass
    

    The benchmarks themselves will automatically handle the missing features if the above is done.

  • leave the timing code to ASV -- don't implement your own

  • the benchmarks are written as Python modules in the benchmark directory (and subdirectories thereof). There are no formal naming requirements for these modules. Benchmarks are generally written as functions of the form time_feature() with the time prefix, sometimes within a broader class object. A setup method or attribute may be used to perform operations that are required for the test suite, but should not be included in the performance timing.

  • parametrized benchmarks can be quite powerful for testing several inputs to a test; note that all possible combinations will be tested, and it is very useful to label the parameters with names as these will be nicely summarized in the output:

    params = (['XTC', 'TRR', 'DCD'],
              [10, 20, 30])
    param_names = ['traj_format', 'num_atoms']
    
  • memory usage can also be profiled with test prefixes including mem and peakmem

Advanced Notes

  • the depedencies installed in the clean conda benchmarking environments, and indeed the decision to use conda over virtualenv, can be controlled in the asv.conf.json file, as can which versions of Python are probed
  • the above file also controls which branch of MDAnalysis is used for a first-pass check of the benchmarks that are written--regardless of where you run the benchmarks from, the current JSON file indicates that ASV will check your benchmark code against the latest commit hash on develop branch first, before running the actual benchmarks for the specified commit hashes