Visual testing

We use jest-image-snapshot for a simple suite of visual regression tests (VRT). Note, that the current system is a work in progress and may change in the future.

Set up

The suite will pick up and run any test file with the *.vrt.js extension. By default we turn every storybook scenario file into a single snapshot test. To add a new test, simply create a new .scenario.js file in the __tests__ directory of a component.

Note, a temporary requirement is that the export name and file name of the scenario are identical.

You can configure each scenario by adding a property matching the scenario name to the vrt config object in vrt/config.js.

// vrt/config.js
module.exports = {
  'modal': {
    // configuration for `modal` scenario
  }
}

The possible properties are:

1. skip: Skip this scenario. Default is false.
2. interactions: An array of interaction configuration objects. Default is [].

So what does an interaction configuration object look like?

// vrt/config.js
module.exports = {
  // ...
  'input-password': {
    interactions: [
      {
        name: 'togglesMask',
        behavior: async page => {
          const toggleSelector = `[data-e2e="mask-toggle"]`;
          await page.$(toggleSelector);
          await page.click(toggleSelector);
        },
      },
    ],
  },
}

In this example we are adding interactions via the interactions property to the input-password scenario. In the specified behavior we click the password input's toggle mask button to verify that the text properly unmasks.

The interaction configuration object requires two properties:

1. A name, which will be appended to the snapshot file name for identification.
2. A behavior, which is an async function that is passed page, a Puppeteer Page instance. Use page to arrange the scenario however you like. The actual snapshot will be generated after this behavior function resolves.

The interactions array can consist of multiple interaction configuration objects. Each object corresponds to a single new snapshot test (in addition to default snapshot). The snapshot in the above example is saved in our snapshot directory as input-password__togglesMask.png.

Docker

The visual regression tests are run within a Docker container so you will have inconsistent results with our saved snapshots if you do not run the tests in Docker.

To run the tests locally you can use docker-compose:

To prepare the visual tests:

$ docker-compose down    # ensure prior services are not running
$ docker-compose build   # sadly required after any change to source

The first run of docker-compose build may take a little while. Subsequent runs should go faster.

To run the visual tests:

$ yarn vrt:docker:run

To update the visual test snapshots:

$ yarn vrt:docker:update

Note, you can still run the test suite locally outside of Docker for debugging purposes with yarn vrt, but the official snapshots must be generated and updated with Docker. CI will not pass otherwise.

Any diffs generated by the test run will be put in __artifacts__. The snapshots themselves are stored in vrt/__image_snapshots__.

CI

If a commit is part of a pull request, the vrt job will run the snapshot tests with the update flag enabled. If there are any changes, a new PR will be automatically opened with a --vrt suffix. This new snapshot PR should be linked to in the original PR. If the changes on that branch seem okay the original author may merge the changes into their branch.

This is the preferred way to update snapshots so that folks do not need to worry about Docker locally. You can still update snapshots locally - but we provide a convenient async way to do it as well.

Note, if the commit triggering CI is not part of a pull request, the vrt job will run the snapshot tests with out the update flag. This means it will just test to see if there are any detectable visual changes. If there are, the diffs will be uploaded as artifacts in Buildkite.