tests: reorganize unit and integration suites to make them easier to…

… split for travis (canonical#1638)

.travis.yml

@@ -17,23 +17,23 @@ jobs:
       script: sudo ./tools/travis/run_tests.sh static
     - stage: unit
       if: type != cron
-      script: sudo ./tools/travis/run_tests.sh unit
+      script: sudo ./tools/travis/run_tests.sh snapcraft/tests/unit
     - if: type != cron
-      script: SNAPCRAFT_TEST_MOCK_MACHINE=armv7l sudo ./tools/travis/run_tests.sh unit
+      script: SNAPCRAFT_TEST_MOCK_MACHINE=armv7l sudo ./tools/travis/run_tests.sh snapcraft/tests/unit
     - stage: snap
       if: type != cron
       script: sudo ./tools/travis/build_snapcraft_snap.sh
     - stage: integration
       if: type != cron
-      script: sudo ./tools/travis/run_tests.sh integration
+      script: sudo ./tools/travis/run_tests.sh snapcraft/tests/integration/general
     - if: type != cron
-      script: sudo ./tools/travis/run_tests.sh plugins
+      script: sudo ./tools/travis/run_tests.sh snapcraft/tests/integration/plugins
     - if: type != cron
-      script: sudo ./tools/travis/run_tests.sh store
+      script: sudo ./tools/travis/run_tests.sh snapcraft/tests/integration/store
     - if: type != cron
-      script: sudo ./tools/travis/run_tests.sh containers
+      script: sudo ./tools/travis/run_tests.sh snapcraft/tests/integration/containers
     - if: type != cron
-      script: sudo ./tools/travis/run_tests.sh snapd
+      script: sudo ./tools/travis/run_tests.sh snapcraft/tests/integration/snapd
     # CLA check, only in pull requests, not comming from the bot.
     - if: type = pull_request AND sender != snappy-m-o
       script: ./tools/travis/run_cla_check.sh

TESTING.md

@@ -27,22 +27,22 @@ The static tests suite performs a static analysis on the source code without exe
 
 The unit tests is a suite of low-level white box tests. They excercise units of code to verify that the different parts work as expected in a fully isolated way. Ideally, these tests should call only public functions, objects and methods, leaving the internals of snapcraft as implementation details that can change without having to modifying any tests. To isolate the units from their environment and dependencies we can replace those with test doubles. In order to set up test doubles, we prefer dependency injection through arguments than excessive mocking. These tests can verify the results checking the output printed to the command line, checking the files created during the excecution, inspecting the calls made to the test doubles and verifying that the expected exceptions were thrown.
 
-These tests are in the `snapcraft/tests` directory.
+These tests are in the `snapcraft/tests/unit` directory.
 
 ### Integration tests
 
 The integration tests are a group of suites that excercise snapcraft as a black box. They are only allowed to set up the environment where snapcraft runs and create files; but for the execution phase of the test they can only run the snapcraft command or one of its subcommands. To verify the results they can check the output printed to the command line, the return value of the snapcraft command, and any files created during the execution.
 
-This suite was split in four: plugins, store, snapd and other integration tests. This split is artificial, we made it just because the full suite takes more time than what Travis allows for a single job.
-
-These tests are in the `integration_tests` directory, with the `snapcraft.yamls` and other source files for the tests snaps in `integration_tests/snaps`.
+These tests are in the `snapcraft/tests/integration` directory, with the `snapcraft.yamls` and other source files for the tests snaps in `snapcraft/tests/integration/snaps`.
 
 ### Slow tests
 
 Some tests take too long. This affects the pull requests because we have to wait for a long time, and they will make Travis CI timeout because we have only 50 minutes per suite in there. The solution is to tag these tests as slow, and don't run them in all pull requests. These tests will only be run in autopkgtests.
 
 To mark a test case as slow, set the class attribute `slow_test = True`.
 
+To run all the tests, including the slow ones, set the environment variable `SNAPCRAFT_SLOW_TESTS=1`.
+
 ### Snaps tests
 
 The snaps tests is a suite of high-level tests that try to simulate real-world scenarios of a user interacting with snapcraft. They cover the call to snapcraft to generate a snap file from the source files of a fully functional project, the installation of the resulting snap, and the execution of the binaries and services of this snap.
@@ -59,50 +59,52 @@ Then, you'll need a few more dependencies:
 
 ### Running the tests
 
-To run all the tests execute:
+To run the static tests, execute:
 
-    ./runtests.sh
+    ./runtests.sh static
 
-You can selectively run only one of the suites, and apply a filter:
+To run the unit tests, execute:
 
-    ./runtests.sh [static|unit|integration|plugins|store|snaps] [pattern]
+    ./runtests.sh snapcraft/tests/unit
 
-Examples:
+To run the integration tests, execute:
 
-  * To run only the static tests:
+    ./runtests.sh snapcraft/tests/integration
 
-    ```
-    ./runtests.sh static
-    ```
+You can also run a subsuite of the unit or integration suites specifying the path to the directory.
+For example:
 
-  * To run only the full unit test suite:
+  * To run only the unit tests for the plugins:
 
     ```
-    ./runtests.sh unit
+    ./runtests.sh snapcraft/tests/unit/plugins
     ```
 
-  * To run only the unit tests in the file called `test_init.py`:
+  * To run only the integration tests for the store:
 
     ```
-    ./runtests.sh unit test_init.py
+    ./runtests.sh snapcraft/tests/integration/store
     ```
 
-  * To run only the integration tests that interact with the store:
+And you can also run a single test module, test case or test function using the usual python way.
+For example:
+
+  * To run only the unit tests in the test_nodejs module:
 
     ```
-    ./runtests.sh store
+    python3 -m unittest snapcraft.tests.unit.plugins.tests_nodejs
     ```
 
-  * To run only the integration tests for the python plugin:
+  * To run only the unit tests in the NodePluginTestCase:
 
     ```
-    ./runtests.sh plugins *python*
+    python3 -m unittest snapcraft.tests.unit.plugins.tests_nodejs.NodePluginTestCase
     ```
 
-  * To run the integration tests that are not related to plugins or the store:
+  * To run only the unit test named test_pull_executes_npm_run_commands:
 
     ```
-    ./runtests.sh integration
+    python3 -m unittest snapcraft.tests.unit.plugins.tests_nodejs.NodePluginTestCase.test_pull_executes_npm_run_commands
     ```
 
 The snaps tests script has more complex arguments. For an explanation of them, run:
@@ -111,11 +113,11 @@ The snaps tests script has more complex arguments. For an explanation of them, r
 
 The integration and snaps suites can be run using the snapcraft source from the repository, or using the snapacraft command installed in the system. By default, they will use the source code, so you can modify your clone of the repository and verify that your changes are correct. If instead you want to verify that the snapcraft version installed in your system is correct, run them with the environment variable `SNAPCRAFT_FROM_INSTALLED` set, like this:
 
-    SNAPCRAFT_FROM_INSTALLED=1 ./runtests.sh [integration|plugins|store|snaps] [pattern]
+    SNAPCRAFT_FROM_INSTALLED=1 ./runtests.sh snapcraft/tests/integration
 
 The store tests by default will start fake servers that are configured to reply like the real store does. But you can run them also against the staging and production store servers. To do that, you will need to set the `TEST_STORE` environment variable to either `staging` or `production`, and you also have to pass credentials for a valid user in that store with the environment variable `TEST_USER_EMAIL` and `TEST_USER_PASSWORD`, like this:
 
-    TEST_STORE=staging [email protected] TEST_USER_PASSWORD=Hola123* ./runtests.sh store [pattern]
+    TEST_STORE=staging [email protected] TEST_USER_PASSWORD=Hola123* ./runtests.sh snapcraft/tests/integration/store
 
 ## Autopkgtests for the snapcraft deb
 

debian/tests/integrationtests

@@ -5,7 +5,7 @@ set -e
 echo 'ubuntu ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers.d/autopkgtest
 
 if [ -z "$SNAPCRAFT_AUTOPKGTEST_SUITES" ]; then
-    suites="integration_tests integration_tests/store integration_tests/plugins integration_tests/snapd"
+    suites="snapcraft/tests/integration"
 else
     suites=$SNAPCRAFT_AUTOPKGTEST_SUITES
 fi

debian/tests/integrationtests-general

@@ -3,4 +3,4 @@
 set -e
 
 script_path="$(dirname "$0")"
-SNAPCRAFT_AUTOPKGTEST_SUITES=integration_tests $script_path/integrationtests
+SNAPCRAFT_AUTOPKGTEST_SUITES=snapcraft/tests/integration/general $script_path/integrationtests

debian/tests/integrationtests-plugins

@@ -3,4 +3,4 @@
 set -e
 
 script_path="$(dirname "$0")"
-SNAPCRAFT_AUTOPKGTEST_SUITES=integration_tests/plugins $script_path/integrationtests
+SNAPCRAFT_AUTOPKGTEST_SUITES=snapcraft/tests/integration/plugins $script_path/integrationtests

debian/tests/integrationtests-snapd

@@ -3,4 +3,4 @@
 set -e
 
 script_path="$(dirname "$0")"
-SNAPCRAFT_AUTOPKGTEST_SUITES=integration_tests/snapd $script_path/integrationtests
+SNAPCRAFT_AUTOPKGTEST_SUITES=snapcraft/tests/integration/snapd $script_path/integrationtests

debian/tests/integrationtests-store

@@ -3,4 +3,4 @@
 set -e
 
 script_path="$(dirname "$0")"
-SNAPCRAFT_AUTOPKGTEST_SUITES=integration_tests/store $script_path/integrationtests
+SNAPCRAFT_AUTOPKGTEST_SUITES=snapcraft/tests/integration/store $script_path/integrationtests

runtests.sh

@@ -21,110 +21,40 @@ export PATH=$(pwd)/bin:$PATH
 export PYTHONPATH=$(pwd)${PYTHONPATH:+:$PYTHONPATH}
 
 parseargs(){
-    if [[ "$#" -eq 0 ]] || [[ "$1" == "all" ]]; then
-        export RUN_STATIC="true"
-        export RUN_UNIT="true"
-        export RUN_INTEGRATION="true"
-        export RUN_STORE="true"
-        export RUN_PLUGINS="true"
-        export RUN_CONTAINERS="true"
-        export RUN_SNAPD="true"
-        export RUN_SNAPS="true"
-        export RUN_SPREAD="true"
+    if [[ "$#" -eq 0 ]]; then
+        echo "Usage: ./runtests.sh unit|snaps|<snapcraft-suite>"
+        echo "<snapcraft-suite> can be snapcraft/tests, or any of its subdirectories."
+        exit 1
     else
         if [ "$1" == "static" ] ; then
-            export RUN_STATIC="true"
-        elif [ "$1" == "unit" ] ; then
-            export RUN_UNIT="true"
-        elif [ "$1" == "integration" ] ; then
-            export RUN_INTEGRATION="true"
-        elif [ "$1" == "store" ] ; then
-            export RUN_STORE="true"
-        elif [ "$1" == "plugins" ] ; then
-            export RUN_PLUGINS="true"
-        elif [ "$1" == "containers" ] ; then
-            export RUN_CONTAINERS="true"
-        elif [ "$1" == "snapd" ] ; then
-            export RUN_SNAPD="true"
+            run_static_tests
         elif [ "$1" == "snaps" ] ; then
-            export RUN_SNAPS="true"
-        # Temporary: backward compatibility until CI run the "snaps" target
-        elif [ "$1" == "examples" ] ; then
-            export RUN_SNAPS="true"
+            # shift to remove the test suite name and be able to pass the rest
+            # to the snaps suite.
+            shift
+            run_snaps "$@"
         elif [ "$1" == "spread" ] ; then
-            export RUN_SPREAD="true"
+            run_spread
         else
-            echo "Not recognized option, should be one of all, static, unit, integration, store, plugins, containers, snapd, snaps or spread"
-            exit 1
+            run_snapcraft_tests "$@"
         fi
     fi
 }
 
 python3 -m coverage 1>/dev/null 2>&1 && coverage="true"
 
 run_static_tests(){
-    SRC_PATHS="bin snapcraft integration_tests snaps_tests external_snaps_tests setup.py"
+    SRC_PATHS="bin snapcraft snaps_tests external_snaps_tests setup.py"
     python3 -m flake8 --max-complexity=10 $SRC_PATHS
 }
 
-run_unit_tests(){
-    if [[ "$#" -lt 2 ]]; then
-        pattern="test_*.py"
-    else
-        pattern=$2
-    fi
-
-    if [ ! -z "$coverage" ]; then
+run_snapcraft_tests(){
+    if [[ ! -z "$coverage" ]] && [[ "$1" == "snapcraft/tests/unit"* ]]; then
         python3 -m coverage erase
-        python3 -m coverage run --branch --source snapcraft -m unittest discover -b -v -s snapcraft/tests -t . -p $pattern
-    else
-        python3 -m unittest discover -b -v -s snapcraft/tests -t . -p $pattern
-    fi
-}
-
-run_integration(){
-    if [[ "$#" -lt 2 ]]; then
-        pattern="test_*.py"
-    else
-        pattern=$2
-    fi
-    python3 -m unittest discover -b -v -s integration_tests -p $pattern
-}
-
-run_store(){
-    if [[ "$#" -lt 2 ]]; then
-        pattern="test_*.py"
-    else
-        pattern=$2
-    fi
-    python3 -m unittest discover -b -v -s integration_tests/store -p $pattern
-}
-
-run_plugins(){
-    if [[ "$#" -lt 2 ]]; then
-        pattern="test_*.py"
-    else
-        pattern=$2
-    fi
-    python3 -m unittest discover -b -v -s integration_tests/plugins -p $pattern
-}
-
-run_containers(){
-    if [[ "$#" -lt 2 ]]; then
-        pattern="test_*.py"
-    else
-        pattern=$2
-    fi
-    python3 -m unittest discover -b -v -s integration_tests/containers -p $pattern
-}
-
-run_snapd(){
-    if [[ "$#" -lt 2 ]]; then
-        pattern="test_*.py"
+        python3 -m coverage run --branch --source snapcraft -m unittest discover -b -v -s "$1" -t .
     else
-        pattern=$2
+        python3 -m unittest discover -b -v -s "$1" -t .
     fi
-    python3 -m unittest discover -b -v -s integration_tests/snapd -p $pattern
 }
 
 run_snaps(){
@@ -142,63 +72,13 @@ run_spread(){
 
 parseargs "$@"
 
-if [ ! -z "$RUN_STATIC" ] ; then
-    run_static_tests
-fi
-
-if [ ! -z "$RUN_UNIT" ]; then
-    run_unit_tests "$@"
-fi
-
-if [ ! -z "$RUN_INTEGRATION" ]; then
-    run_integration "$@"
-fi
-
-if [ ! -z "$RUN_STORE" ]; then
-    run_store "$@"
-fi
-
-if [ ! -z "$RUN_PLUGINS" ]; then
-    run_plugins "$@"
-fi
-
-if [ ! -z "$RUN_CONTAINERS" ]; then
-    run_containers "$@"
-fi
-
-if [ ! -z "$RUN_SNAPD" ]; then
-    run_snapd "$@"
-fi
+if [[ ! -z "$coverage" ]] && [[ "$1" == "snapcraft/tests/unit"* ]]; then
+    python3 -m coverage report
 
-if [ ! -z "$RUN_SNAPS" ]; then
-    if [ "$1" == "snaps" ] ; then
-        # shift to remove the test suite name and be able to pass the rest
-        # to the snaps suite.
-        shift
-    fi
-    ## Temporary: backward compatibility until CI run the "snaps" target
-    if [ "$1" == "examples" ] ; then
-        # shift to remove the test suite name and be able to pass the rest
-        # to the snaps suite.
-        shift
-    fi
-    ##
-    run_snaps "$@"
-fi
-
-if [ ! -z "$RUN_SPREAD" ]; then
-    run_spread
-fi
-
-if [ ! -z "$RUN_UNIT" ]; then
-    if [ ! -z "$coverage" ]; then
-        python3 -m coverage report
-
-        echo
-        echo "Run 'python3-coverage html' to get a nice report"
-        echo "View it by running 'x-www-browser htmlcov'"
-        echo
-    fi
+    echo
+    echo "Run 'python3-coverage html' to get a nice report"
+    echo "View it by running 'x-www-browser htmlcov'"
+    echo
 fi
 
 echo -e "\e[1;32mEverything passed\e[0m"

setup.py

@@ -149,5 +149,5 @@
             'requests',
             'libarchive-c',
         ],
-        test_suite='snapcraft.tests',
+        test_suite='snapcraft.tests.unit',
     )