README

Android JUnit Report Test Runner
================================

Introduction
------------

The Android JUnit report test runner is a custom instrumentation test
runner for Android that creates XML test reports.  These reports are
in a similar format to those created by the Ant JUnit task's XML
formatter, allowing them to be integrated with tools that support that
format (e.g. continuous integration servers).

How It Works
------------

This test runner may be used as a drop-in replacement for the standard
InstrumentationTestRunner provided in the Android SDK.  It is in fact
an extension of that runner, supporting the same functionality with
the addition of XML report generation.

To be consistent with existing SDK support for generating coverage
reports, this runner outputs its XML report in the file storage area
for the application under test.  By default, the report file is
produced at:

/data/data/<test application package>/files/junit-report.xml

Note, however, that sometimes the android system fails to create the
files directory, in which case the runner will fall back to:

/data/data/<application under test package>/files/junit-report.xml

The file may be retrieved from the device using adb pull (see below
for more details).  You may also customise the output location,
file name and even choose to produce multiple output files (one per
suite).  Again, details may be found below.

Using the Runner
----------------

OBTAINING THE JAR

First you will need to obtain a release jar file containing the
runner.  You can build from source (see below), but it's probably
easier to just grab the latest jar from:

http://github.com/jsankey/android-junit-report/downloads

You can also grab the latest directly from the build server:

http://pulse.zutubi.com/browse/projects/android-junit-report-release/builds/success/

(Click on "jar" in the "featured artifacts" table on the right of the
page.)

USING WITH ANT BUILDS

To use this runner with an Ant build (based on the Android SDK support
for Ant):

  * Add android-junit-report-<version>.jar to your test project's
    libraries by dropping the jar into a libs/ subdirectory.

  * Update your test project's manifest <instrumentation> tag.  Change
    the android:name attribute's value to:

    com.zutubi.android.junitreport.JUnitReportTestRunner

  * Edit your build.xml file and set the test.runner property to:

    com.zutubi.android.junitreport.JUnitReportTestRunner

    This must be set *before* running the Android <setup/> task.

These steps will cause your tests to be executed with the custom
runner, which will produce the junit-report.xml file in the storage
area for the project under test.  To retrieve the file from the
device, you can use Ant to run an adb pull, for example:

    <target name="fetch-test-report">
        <echo>Downloading XML test report...</echo>
        <mkdir dir="${reports.dir}"/>
        <exec executable="${adb}" failonerror="true">
	    <arg line="${adb.device.arg}"/>
            <arg value="pull" />
            <arg value="/data/data/${manifest.package}/files/junit-report.xml" />
            <arg value="${reports.dir}/junit-report.xml" />
        </exec>
    </target>

USING WITH ECLIPSE

When running within Eclipse, there is not much call for XML test
reports.  However, assuming you have edited your test project's
manifest to specify this custom runner (as described above), you may
also need to tell Eclipse to use this runner to avoid problems.  You
can do this by:

  * Adding android-junit-report-<version>.jar to the build path of
    your test project in Eclipse.
  * Ensuring all existing run configurations for unit tests specify
    an Instrumentation runner of:

    com.zutubi.android.junitreport.JUnitReportTestRunner

If you see an error about not being able to find a target package for
android.test.InstrumentationTestRunner, it is likely you have an
existing run configuration that is set to use the default runner.  You
can recreate any such configurations and the issue will disappear.

Report Format
-------------

As stated above, the XML report format is based on the Ant JUnit
task's XML formatter.  A few caveats apply:

  * In the default single file mode, multiple suites are all placed in
    a single file under a root <testsuites> element.  In multiple file
    mode each XML file contains a single suite.
  * Redundant information about the number of nested cases within a
    suite is omitted.
  * Durations are omitted from suites.
  * Neither standard output nor system properties are included.

The differences mainly revolve around making this reporting as
lightweight as possible. The report is streamed as the tests run,
making it impossible to, e.g. include the case count in a
<testsuite> element.

The format is intended to be "compatible enough" for integration with
existing tools.  In my particular case I use the reports with my own
Pulse Continuous Integration Server, so compatibility with it was my
first target.  If you find an incompatibility with another tool, let
me know (see below) and I will see what can be done.

Customising Via Arguments
-------------------------

The runner supports the following arguments:

  * multiFile: if set to true, a new report file is generated for each
    test suite.  If false, a single report is generated containing all
    suites.  Defaults to false.
  * reportFile: the name of the report file to generate (single
    file mode) or a pattern for the name of the files to generate
    (multiple file mode).  In the latter case the string $(suite)
    will be substituted with the test suite name to produce the file
    name for each suite.  See the reportDir argument for the
    destination of the report files.  Defaults to junit-report.xml in
    single file mode, junit-report-$(suite).xml in multiple file
    mode.
  * reportDir: if specified, the absolute path to a directory in which
    to write the report file(s).  The specified directory must exist
    and be accessible to the test application.  If not specified files
    are written to the test application's data area if possible, or the
    application under test's data area if that fails (which can happen
    for unknown reasons in some environments).  Defaults to
    unspecified.
  * filterTraces: if true, stack traces in the report will be filtered
    to remove common noise (e.g. framework methods).  Defaults to
    true.

To specify arguments, use the -e flag to adb shell am instrument, for
example:

adb shell am instrument -w -e reportFile my-report.xml \
  com.example.test/com.zutubi.android.junitreport.JUnitReportTestRunner

If you need to pass arguments in your Ant build, you must override
the built-in run-tests target.  As an example, if you prefer multiple
output files (which are more compatible with some tools), you need to
set the multiFile argument to true.  In this case you will also need
to pull the whole directory of files from the device after running
your tests:

    <target name="run-tests" depends="-install-tested-project, install"
            description="Runs tests from the package defined in test.package property">
        <property name="reports.dir" value="${out.dir}/reports"/>
        <property name="files.dir" value="/data/data/${manifest.package}/files"/>

        <echo>Cleaning up previous test reports...</echo>
        <delete dir="${reports.dir}"/>
        <exec executable="${adb}" failonerror="true">
            <arg line="${adb.device.arg}" />
            <arg value="shell" />
            <arg value="rm" />
            <arg value="${files.dir}/*" />
        </exec>

        <echo>Running tests...</echo>
        <exec executable="${adb}" failonerror="true">
            <arg line="${adb.device.arg}"/>
            <arg value="shell" />
            <arg value="am" />
            <arg value="instrument" />
            <arg value="-w" />
            <arg value="-e" />
            <arg value="coverage" />
            <arg value="@{emma.enabled}" />
            <arg value="-e" />
            <arg value="multiFile" />
            <arg value="true" />
            <arg value="${manifest.package}/${test.runner}" />
        </exec>
        
        <echo>Downloading XML test reports...</echo>
        <mkdir dir="${reports.dir}"/>
        <exec executable="${adb}" failonerror="true">
            <arg line="${adb.device.arg}"/>
            <arg value="pull" />
            <arg value="${files.dir}" />
            <arg value="${reports.dir}" />
        </exec>
    </target>

Building From Source
--------------------

If you would like to modify the runner, or build it yourself for any
other reason, you will need:

  * A JDK, version 1.5 or later.
  * The Android SDK (or at least a stub android.jar as provided in the
    SDK).
  * Apache Ant version 1.7 or later.

To run a build:

  * Create a file local.properties in the directory containing this
    README.  In this file, define the location of an android.jar to
    build against, for example:

    android.jar=/usr/local/java/android/platforms/android-8/android.jar

    where /usr/local/java/android is the root of an Android SDK.

  * Run ant in this same directory:

    $ ant

The jar will be created at build/android-junit-report-dev.jar.

Contact
-------

If you have any thoughts, questions etc about the runner, you can
contact me at:

  jason@zutubi.com

All feedback is welcome.

License
-------

This code is licensed under the Apache License, Version 2.0.  See the
LICENSE file for details.