tmt

Test Management Tool

Description

The tmt python module and command line tool implement the L1 and L2 Metadata Specification which allows to store all needed test execution data directly within a git repository. In this way it makes tests independent on any external test management system.

The Flexible Metadata Format fmf is used to store data in both human and machine readable way close to the source code. Thanks to inheritance and elasticity metadata are organized in the structure efficiently, preventing unnecessary duplication.

Command line tool allows to easily create new tests, convert old metadata, list and filter available tests and verify them against the L1 specification. Plans are used to group tests and precisely define individual test steps defined by the L2 specification, like environment preparation. Stories are used to track implementation, test and documentation coverage for individual features.

Last but not least, the tool provides a user-friendly way how to run, debug and develop tests directly from your laptop across many different test environments. This is currently a proof-of-concept so many features are still on the way. Check stories to see which functionality has already been implemented.

Synopsis

Command line usage is straightforward:

tmt command [options]

Examples

Run all or selected steps for each plan:

tmt run
tmt run discover
tmt run prepare execute

List tests, show details, check against the specification:

tmt test ls
tmt test show
tmt test lint

Create a new test, convert old metadata:

tmt test create
tmt test convert

List plans, show details, check against the specification:

tmt plan ls
tmt plan show
tmt plan lint

List stories, check details, show coverage status:

tmt story ls
tmt story show
tmt story coverage

Many commands support regular expression filtering and other specific options:

tmt story ls cli
tmt story show create
tmt story coverage --implemented

Check help message of individual commands for the full list of available options.

Options

Here is the list of the most frequently used commands and options.

Run

The run command is used to execute test steps. By default all test steps are run. See the L2 Metadata specification for detailed description of individual steps. For now here is at least a brief overview:

discover

gather and show information about test cases to be executed

provision

provision an environment for testing (or use localhost)

prepare

configure environment for testing (e.g. ansible playbook)

execute

run the tests (using the specified framework and its settings)

report

provide an overview of test results and send notifications

finish

additional actions to be performed after the test execution

Note: This is only preview / draft of future functionality. Features described above are not implemented yet.

Test

Manage tests (L1 metadata). Check available tests, inspect their metadata, gather old metadata from various sources and stored them in the new fmf format.

ls

List available tests.

show

Show test details.

lint

Check tests against the L1 metadata specification.

create

Create a new test based on given template.

convert

Convert old test metadata into the new fmf format.

Plan

Manage test plans (L2 metadata). Search for available plans. Explore detailed test step configuration.

ls

List available plans.

show

Show plan details.

lint

Check plans against the L2 metadata specification.

Story

Manage user stories. Check available user stories. Explore coverage (test, implementation, documentation).

ls

List available stories.

show

Show story details.

coverage

Show code, test and docs coverage for given stories.

export

Export selected stories into desired format.

Utils

Various utility options.

--path PATH	Path to the metadata tree (default: current directory)
--verbose	Print additional information standard error output
--debug	Turn on debugging output, do not catch exceptions

Check help message of individual commands for the full list of available options.

Install

The tmt package is available in Fedora and EPEL:

dnf install tmt

Install the latest version from the Copr repository:

dnf copr enable psss/tmt
dnf install tmt

Use PIP (you can omit the --user flag if in a virtualenv):

pip install --user tmt

Develop

In order to experiment, play with the latest bits and develop improvements it is best to use a virtual environment:

mkvirtualenv tmt
git clone https://github.com/psss/tmt
cd tmt
pip install -e .

Install python3-virtualenvwrapper to easily create and enable virtual environments using mkvirtualenv and workon.

Links

Git: https://github.com/psss/tmt

Docs: http://tmt.readthedocs.io/

Stories: https://tmt.readthedocs.io/en/latest/stories.html

Issues: https://github.com/psss/tmt/issues

Releases: https://github.com/psss/tmt/releases

Copr: http://copr.fedoraproject.org/coprs/psss/tmt

PIP: https://pypi.org/project/tmt/

Travis: https://travis-ci.org/psss/tmt

Coveralls: https://coveralls.io/github/psss/tmt

Specification: https://pagure.io/fedora-ci/metadata

Flexible Metadata Format: http://fmf.readthedocs.io/

Packit & Testing Farm: https://packit.dev/testing-farm/

Authors

Petr Šplíchal, Miro Hrončok, Alexander Sosedkin, Lukáš Zachar, Petr Menšík, Leoš Pol and Miroslav Vadkerti.

Copyright

Copyright (c) 2019 Red Hat, Inc.

This program is free software; you can redistribute it and/or modify it under the terms of the MIT License.