This repository contains build acceptance tests used to evaluate the functional correctness of content assembled into a distribution based on groups of packages.
To run individual test files level, execute bats -t ./<test_file_name>
while changed into the working directory of the test file. When executing
test files this way, results will be logged to the standard output.
All test results are formatted according to the Test Anything Protocol (TAP). This was chosen to allow all build and release systems work with test results in a single format.
When writing a test, it is important to fully define all of the requirements for running the test properly. Answering the following questions defines the necessary requirements:
- What bundles are needed?
- How much disk space is needed?
- How much RAM and CPU is needed?
- What other characteristics of the environment are required?
After defining all testing requirements, tests may be written. Tests for this
repository are written to execute with bats
. The
README.md for
bats
descirbes how to write tests. Conventionally, tests written to execute
with bats
are saved in a test file with a .bats
extension. However, this
repository names test files with a .t
extension. Changing the test file
extension does not affect execution.
While not recommended, depending on the test process, a test file may be a bash
script without any bats
tests in it. While bats
is the usual TAP producer,
any shell script capable of producing TAP results may be constructed to achieve
more complex integration. This may include calling out to bats
again,
because it is a TAP producer.
Relevant files:
- requirements
- Defines the bundles being tested and the bundles required for proper test execution
- Used by
run-bbt.sh
to determine if an environment has the required bundles installed and can run tests
- clr-installer.yaml
- Defines a raw image
- Used by clr-installer to provision a virtual environment for running tests
Tests will need some number of bundles to execute properly. The smallest
possible bundle list should be defined in the requirements
file. This
bundle list should not include the standard os-core
, os-core-update
,
os-testsuite
, and kernel-*
bundles required for any environment to operate
and run tests. This same bundle list must also be present in the Bundles
section of the clr-installer.yaml file.
Relevant files:
- clr-installer.yaml
Tests will need some amount of disk space during exeution. The virtual
environment will need some amount of disk space for the bundles listed in the
requirements
file in addition to that needed by the standard bundles required
for any environment to operate and run tests. A general rule of thumb to
follow:
space needed during test execution + 2 * space needed for all bundles = total space
The multiplier is required to allow for a fast install of content into the raw image file with swupd. The swupd state directory can be placed inside the image and content may be hardlinked to avoid a slow copy over a partition boundary. The end result is a reduction in image creation time.
Relevant files:
- clear.xml
- Defines the full domain description of a virtual machine
- Used by
libvirt
to start a virtual machine
Tests will need some amount of RAM and CPU in order to execute correctly. Under provisioning resources to a virtual machine may cause tests to falsely fail. Over provisioning creates competition for resources as multiple virtual machines may be executed on a single server.
Relevant files:
- labels
- CSV format:
<label>,<test_file_name>
- Used by
run-bbt.sh
to limit which tests get run
- CSV format:
Tests will need executed in particular contexts. A test may need run
physically, but not virtually. Or, a test may need run virtually, but not
physically. Still yet, a test may require specialized hardware support. To
identify which contexts in which a test needs to run, a label describing the
context can be created with a label-to-test file name mapping in the labels
file. A test file may show up multiple times but under different labels, or
for only one label. Each line in the labels
file can only contain one
label-to-test file name mapping.
The main labels that exist are:
- virtual
- Usage requires the presence of
clr-installer.yaml
andclear.xml
files - Tests will run in a full virtual environment defined by the relevant files
- Usage requires the presence of
- physical
- A subset of tests that are important to run on a baremetal system
- Only use for when virtual validation is not enough
If specialized hardware support is required, a new label indicating the
specific hardware may be added to the labels
file. To have tests requiring
specialized hardware support run regularly, engage with the DevOps and QA team
for provisioning.
*OTHER NAMES AND BRANDS MAY BE CLAIMED AS THE PROPERTY OF OTHERS