GitLab QA - End-to-end tests for GitLab

This directory contains end-to-end tests for GitLab. It includes the test framework and the tests themselves.

The tests can be found in qa/specs/features (not to be confused with the unit tests for the test framework, which are in spec/).

Tests use GitLab QA project for environment orchestration in CI jobs.

What is it?

GitLab QA is an end-to-end test framework designed for test GitLab.

These are black-box and entirely click-driven end-to-end tests you can run against any existing instance of GitLab.

How does it work?

1. When we release a new version of GitLab, we build a Docker images for it.
2. Along with GitLab Docker Images we also build and publish GitLab QA images.
3. QA image can be used to execute e2e tests against running instance of GitLab.

Validating GitLab views / partials / selectors in merge requests

We recently added a new CI job that is going to be triggered for every push event in CE and EE projects. The job is called qa:selectors and it will verify coupling between page objects implemented as a part of GitLab QA and corresponding views / partials / selectors in CE / EE.

Whenever qa:selectors job fails in your merge request, you are supposed to fix page objects. You should also trigger end-to-end tests using package-and-qa manual action, to test if everything works fine.

How can I use it?

You can use GitLab QA to exercise tests on any live instance! If you don't have an instance available you can follow the instructions below to use the GitLab Development Kit (GDK). This is the recommended option if you would like to contribute to the tests.

Note that tests are using Chrome web browser by default so it should be installed and present in PATH.

CI

Tests are executed in merge request pipelines as part of the development lifecycle. CI pipeline setup and different types of environments are described in test pipelines documentation page.

Including tests in other projects

Pipeline template for test-on-omnibus E2E tests is designed in a way so it can be included as a child pipeline in other projects.

Minimal configuration example would look like this:

qa-test:
  stage: test
  variables:
    RELEASE: EE
  trigger:
    strategy: depend
    forward:
      yaml_variables: true
      pipeline_variables: true
    include:
      - project: gitlab-org/gitlab
        ref: master
        file: .gitlab/ci/test-on-omnibus/main.gitlab-ci.yml

To set GitLab version used for testing, following environment variables can be used:

• RELEASE: omnibus release, can be string value EE or CE for nightly release of enterprise or community edition of GitLab or can be fully qualified Docker image name
• QA_IMAGE: Docker image of qa code. By default inferred from RELEASE but can be explicitly overridden with this variable

Test specific environment variables

Special GitLab configurations require various specific environment variables to be present for tests to work. These can be provisioned automatically using terraform setup in engineering-productivity-infrastructure project.

Logging

By default tests on CI use info log level. debug level is still available in case of failure debugging. Logs are stored in jobs artifacts.

Writing tests

• Writing tests from scratch tutorial
  • Best practices
  • Using page objects
  • Guidelines
  • Tests with special setup for local environments

Running tests

• Against your GDK environment
• Against GitLab in Docker
• Specific types of tests
• Test configuration

Building a Docker image to test

Once you have made changes to the CE/EE repositories, you may want to build a Docker image to test locally instead of waiting for the gitlab-ce-qa or gitlab-ee-qa nightly builds. To do that, you can run from the top gitlab directory (one level up from this directory):

docker build -t gitlab/gitlab-ce-qa:nightly --file ./qa/Dockerfile ./