Acceptance Testing

This README outlines how to run tests using the system testing framework beaker, specifically for PuppetDB.

Quick Start

The acceptance tests utilise the gem beaker, and are initiated by the rake task rake test:beaker.

An example of how to run the tests:

$ bundle install
$ rake beaker:acceptance

However, there are a number of environment variables that can be utilised to modify the build.

An example of how to run a package based build, with vagrant/virtualbox on Centos 6 would be:

PUPPETDB_INSTALL_TYPE='package' \
PUPPETDB_EXPECTED_RPM_VERSION="1.3.3.73-1" \
rake beaker:acceptance

Source based build:

rake beaker:acceptance

EC2 build from source on Debian 6:

BEAKER_CONFIG="ec2-west-debian6-64mda-64a" \
rake beaker:acceptance

EC2 build with packages on Debian 6:

BEAKER_CONFIG="ec2-west-debian6-64mda-64a" \
PUPPETDB_INSTALL_TYPE='package' \
PUPPETDB_PACKAGE_REPO_URL="http://puppetdb-prerelease.s3.amazonaws.com/puppetdb/1.3.x" \
PUPPETDB_EXPECTED_DEB_VERSION="1.3.3.73-1puppetlabs1" \
rake beaker:acceptance

Note that the tests currently depend on the Puppet internal lein-ezbake project, so running them requires either access to the Puppet VPN during the test run, or a local copy of lein-ezbake. For the latter, run these commands once, while connected to the VPN:

repo='http://nexus.delivery.puppetlabs.net/content/repositories/releases/'
mvn org.apache.maven.plugins:maven-dependency-plugin:2.7:get \
  -Dtransitive=false \
  -DrepoUrl="$repo" \
  -Dartifact=puppetlabs:lein-ezbake:0.2.2 \
  -Ddest=lein-ezbake-0.2.2.jar
mvn org.apache.maven.plugins:maven-dependency-plugin:2.7:get \
  -Dtransitive=false \
  -DrepoUrl="$repo" \
  -Dartifact=puppetlabs:lein-ezbake:0.2.2:pom \
  -Ddest=lein-ezbake-0.2.2.pom

And then this before every beaker:acceptance invocation:

mvn org.apache.maven.plugins:maven-install-plugin:2.5.2:install-file \
  -DpomFile=lein-ezbake-0.2.2.pom \
  -Dfile=lein-ezbake-0.2.2.jar \
  -DlocalRepositoryPath=tmp/m2-local

You can find the required version of ezbake in project.clj.

How to set options

PuppetDB Specific Options

You can set these options in one of two ways; either by specifying them as a key-value pair in the hash that you return from your "--options" file, or by setting environment variables. The symbols listed below are the keys you would put in your hash; the environment variable names are the same but uppercased (shown in parens below, for all of your copy'n'paste needs):

• :puppetdb_install_type (PUPPETDB_INSTALL_TYPE) : Can be one of :git, :pe or :package. This dictates how the software gets installed.

• :install_mode (INSTALL_MODE) : This setting is only relevant when our install_type is set to :package (meaning that we are running the tests against OS packages rather than against source code pulled from git). Legal values are :install and :upgrade; if set to :install, the test will be run against a freshly-installed package built from the latest development source code. If set to :upgrade, we will first install the latest release packages, then start puppetdb, then upgrade to the new package built from development source code.

• :puppetdb_validate_package_version (PUPPETDB_VALIDATE_PACKAGE_VERSION) : This boolean determines whether or not we attempt to verify that the installed package version matches what we expect. This is mostly useful in CI, to make sure we're testing the right code. Legal values are :true and :false.

• :puppetdb_expected_rpm_version (PUPPETDB_EXPECTED_RPM_VERSION) : This is only relevant if :puppetdb_validate_package_version is set to :true. In that case, this setting may contain a version number string (including the rpm release specifier), and the tests will fail on RedHat-based systems if, when we install the development puppetdb package, its version number does not match this string.

• :puppetdb_expected_deb_version (PUPPETDB_EXPECTED_DEB_VERSION) : This is only relevant if :puppetdb_validate_package_version is set to :true. In that case, this setting may contain a version number string, and the tests will fail on Debian-based systems if, when we install the development puppetdb package, its version number does not match this string.

• :puppetdb_use_proxies (PUPPETDB_USE_PROXIES) : This determines whether or not the test run will configure package managers (apt/yum) and maven to use our internal proxy server. This can provide a big speedup for CI, but is not desirable for remote employees during development. Legal values are :true and :false.

• :puppetdb_purge_after_run (PUPPETDB_PURGE_AFTER_RUN) : This determines whether or not the post-suite cleanup phase will remove packages and perform exhaustive cleanup after the run. This is useful if you would like to avoid resetting VMs between every run of the acceptance tests. Defaults to :true.

• :puppetdb_package_build_host (PUPPETDB_PACKAGE_BUILD_HOST) : This specifies the hostname where the final packages built by the packaging job are available. This should typically not need to be overridden, as it defaults to the well-known host name provided by our release engineering team.

• :puppetdb_package_repo_host (PUPPETDB_PACAKGE_REPO_HOST): This specifies the hostname where the final apt/yum repos will be deployed and accessible to the test nodes. When testing under EC2, this will be an s3 "hostname" that differs from our default for internal VMs. This should be overridden by jenkins jobs that are running in EC2.

• :puppetdb_package_repo_url (PUPPETDB_PACKAGE_REPO_URL) : By default, the test setup will install the latest 'master' branch of puppetdb dev packages from puppetlabs.lan; however, if this option is set, then it will try to use the apt/yum repos from that url (appending 'debian', 'el', etc.) instead. This is required for jobs that will be running outside of the puppetlabs LAN.

• :puppetdb_repo_(puppet|facter|hiera) (PUPPETDB_REPO_(PUPPET|FACTER|HIERA)) : Specify the git repository and reference to use for installing Puppet/Facter/Hiera from source. This is primarily so we can test against alternate versions of Puppet, so if the Puppet repo is not specified we fall back to using packages.

• :puppetdb_git_ref (REF) : Specify the specific git ref that the tests should be run against. This should almost always be passed in by the Jenkins job, and not overridden by configuration.

Beaker Specific Options

These options are only environment variables, and are specific to the beaker:acceptance rake task.

###BEAKER_TYPE

Can be one of:

• git: this changes the installation type to use git, and is really associated with FOSS based tests.
• pe: this changes the installation type for pe based tests.

The default is git.

Note: In the future this option will be removed from Beaker.

###BEAKER_CONFIG

This variable allows you to select a host configuration. See the directory acceptance/config/ for a list of availabe configurations (the .cfg extension is not necessary).

###BEAKER_OPTIONS

This allows one to choose from a number of different configuration option files. Including:

• postgres
• embedded_db
• puppet_enterprise

These files are listed in the directory acceptance/options, so you should look at the detail in those files more more information.

###BEAKER_PRESERVE_HOSTS

When this setting is always it will stop the removal of the virtual machine once the task has been completed. onfail will leave them behind only if they have failed. The default is never.

####BEAKER_COLOR

Set this to true to enable color in Beaker. Setting to false will disable color in Beaker.

####BEAKER_XML

Set this to true to produce XML that can be consumed by Jenkins for test reporting.

Extra Information

EC2 Benchmarking

We determined there was a cost/benefit sweet spot by moving to c1.medium from m1.small, the calculations are here:

42 minutes, 6 cents:    m1.small       <-- legacy
24 minutes, 12 cents:   m1.medium
19 minutes, 14.5 cents: c1.medium      <-- good bang for buck
19 minutes, 24 cents:   m1.large
17 minutes, 48 cents:   m1.xlarge
15 minutes, 41 cents:   m2.xlarge      <-- next best jump after c1.medium?
15 minutes, 82 cents:   m2.2xlarge
14 minutes, 50 cents:   m3.xlarge