Skip to content

Latest commit

 

History

History
243 lines (173 loc) · 8.68 KB

TESTING.md

File metadata and controls

243 lines (173 loc) · 8.68 KB

Testing Erlang/OTP

Before you start testing you need to have the Erlang release which you are going to test in your path. See $ERL_TOP/HOWTO/INSTALL.md for instructions on how to build an Erlang release.

Short version

Move to the top directory of the Erlang release you want to test, i.e. cd /ldisk/work/otp

export ERL_TOP=`pwd`
./otp_build setup -a
export PATH=`pwd`/bin:$PATH
./otp_build tests
cd release/tests/test_server
erl -s ts install -s ts run all_tests -s init stop

Where are the tests

There are a lot of tests which test Erlang/OTP (as of 2012 about 12000) and they are located in different places throughout the source tree. Below is a list of the places where you can expect to find tests:

  • $ERL_TOP/lib/AppName/test/
  • $ERL_TOP/erts/test/
  • $ERL_TOP/erts/emulator/test/
  • $ERL_TOP/erts/epmd/test/

Writing tests

All tests are common_test suites and follow the same pattern as all common_test suites. However, a couple of corner cases are handled by a test wrapper called ts. ts allows the test writer to put Makefile.src and Makefile.first files in the data_dir and a special directory called all_SUITE_data.

Makefile.first is run before any other Makefile and is typically used to generate .hrl files which are needed by other test suites. At the moment only the erl_interface tests use this feature and it should remain that way.

Makefile.src is configured to a Makefile using the variables created when configuring the tests. These Makefiles are later run by the test suite to compile whatever platform specific code the tests need to run.

Releasing tests

If you cannot use ct_run in the source tree you have to release the tests into a common test directory. The easiest way to do this is to use otp_build like this:

export ERL_TOP=`pwd`; ./otp_build tests

This will release all tests in Erlang/OTP to $ERL_TOP/release/tests/. If you want to change the directory where the tests are released to use the TESTROOT environmental variable.

In the $TESTROOT you should now see *_test folders. These folders contain everything needed to test Erlang/OTP and are platform independent; if you are testing Erlang on multiple platforms you just have to release on one and copy the tests to all other platforms.

Releasing cross tests

For releasing tests in a cross compilation environment see $ERL_TOP/HOWTO/INSTALL-CROSS.md.

Configuring and Running tests

Running tests is done by first navigating to the $TESTROOT/test_server folder created when you released the tests and then start erl in that directory. The emulator flags specified will be used in the test runs. For example, if you want to test using async threads you have to supply +A 10 to erl when you start it.

To configure and run the tests ts is used. ts is a wrapper module to common_test which takes care of configuration and build issues before common_test is started.

ts has a lot of special options and functions which can be useful when testing Erlang/OTP. For a full listing issue ts:help() in the erlang shell.

Configuring the test environment

Before running released tests you have to install them on the target system. Installing the tests is done by calling ts:install(). in the Erlang shell which you intend to test from. ts:install() is basically a wrapper to a configure script and some Erlang code which figures out what your system looks like and what kind of emulator you are testing with. ts:install() can also take some arguments if necessary, see ts:help() for details.

All variables created by ts:install() are found in $TESTROOT/test_server/variables.

Running the tests

To run all test suites go to $TESTROOT/test_server fire up an Erlang shell and type:

ts:run().

Note that running all tests will require several hours, so you may want to run the test cases for a single application

ts:run(Application, [batch]).

or even part of the test suite for an application, for example

ts:run(emulator, bs, [batch]).

to run all test suite modules starting with bs (i.e. all modules that test the bit syntax).

To run a specific test case in a module, the full name of the module and test case must be specified:

ts:run(emulator, bs_bincomp_SUITE, byte_aligned, [batch]).

Run ts:help(). for more information.

As of R14B02 it is also possibly to start all tests but the erl_interface tests by invoking Common Test directly from the released applications test directory, i.e.

cd $TESTROOT/test_server
$ERL_TOP/bin/ct_run -suite ../compiler_test/andor_SUITE -case t_orelse

Running ct_run from the command line still requires you to do the ts:install() step above.

Convenience for running tests without the release and configuration steps

It can be convenient to run tests with a single command. This way, one do not need to worry about missing to run make release_tests after changing a test suite. The make test command can be used for this purpose. The make test command works when the current directory contains a directory called test and in the root directory of the source code tree.

(Warning) Some test cases do not run correctly or cannot be run at all through the make test command (typically test cases that require test specific C code to be compiled) because make test runs tests directly by invoking the ct_run command instead of using the ts wrapper. One has to follow the procedure described above to run test cases that do not work with make test.

Below are some examples that illustrate how make test can be used:

# ERL_TOP needs to be set correctly
cd /path/to/otp
export ERL_TOP=`pwd`

# Build Erlang/OTP
#
# Note that make test will only compile test code except when
# make test is executed from $ERL_TOP.
./otp_build setup -a

# Run a test case (The ARGS variable is passed to ct_run)
(cd $ERL_TOP/erts/emulator && make ARGS="-suite binary_SUITE -case deep_bitstr_lists" test)

# Run a test suite
(cd $ERL_TOP/lib/stdlib && make ARGS="-suite ets_SUITE" test)

# Run all test suites for an application
(cd $ERL_TOP/lib/asn1 && make test)

# Run all tests
#
# When executed from $ERL_TOP, "make test" will first release and
# configure all tests and then attempt to run all tests with `ts:run`.
# This will take several hours.
(cd $ERL_TOP && make test)

Examining the results

Open the file release/tests/test_server/index.html in a web browser. Or open release/tests/test_server/last_test.html when a test suite is running to examine the results so far for the currently executing test suite (in R14B02 and later you want to open the release/tests/test_server/all_runs.html file to get to the currently running test)

Run tests with Address Sanitizer

First build emulator with asan build target. See $ERL_TOP/HOWTO/INSTALL.md.

Set environment variable ASAN_LOG_DIR to the directory where the error logs will be generated.

export ASAN_LOG_DIR=$TESTROOT/test_server/asan_logs
mkdir $ASAN_LOG_DIR

Set environment variable TS_RUN_EMU to asan.

export TS_RUN_EMU=asan

Then run the tests you want with ts:run as described above. Either inspect the log files directly or use the script at $ERL_TOP/erts/emulator/asan/asan_logs_to_html to read all log files in $ASAN_LOG_DIR and distill them into one html page asan_summary.html. Repeated reports from the same memory leak will for example be ignored by the script and make it easier to analyze.

Run tests with Valgrind

First make sure valgrind is installed, then build OTP from source and build the emulator with valgrind build target. See $ERL_TOP/HOWTO/INSTALL.md.

Set environment variable VALGRIND_LOG_DIR to the directory where the valgrind error logs will be generated.

export VALGRIND_LOG_DIR=$TESTROOT/test_server/vg_logs
mkdir $VALGRIND_LOG_DIR

Set environment variable TS_RUN_EMU to valgrind.

export TS_RUN_EMU=valgrind

Then run the tests you want with ts:run as described above and inspect the log file(s) in $VALGRIND_LOG_DIR.