AndroidGUITesting

Which Factor Impacts GUI Traversal-Based Test Case Generation Technique Most? A Controlled Experiment on Android Applications （QRS2017 best paper）

Our experimentation framework is adapted from the ASE 2015 work of Dr.Shauvik Roy Choudhary@ http://bear.cc.gatech.edu/~shauvik/androtest/.

We owe our thanks to their sharing.

1) Setup Process

1. Download the box file and install VirtualBox，Vagrant；

The box files is shared at https://pan.baidu.com/s/1mAJt9vUD5MHEBP0d9_QwVw passwd：0cff. Or you can email Dr. Bo Jiang: [email protected] to get the big box file.

2. In a terminal, add the Androtest box to Vagrant $ vagrant box add androqrs ./androqrs.box
3. Create a directory, say ~/vagrant/androtest，to host the vagrant machine and inside this directory download this Vagrantfile. This file contains the VirtualBox VM configuration that vagrant uses. Note that the configuration defines 10 VM instances labelled run1-run10. $ mkdir -p ~/vagrant/androtest $ cd ~/vagrant/androtest $ wget https://github.com/gongbell/AndroidGUITesting/blob/master/Vagrantfile
4. You can start the vm using vagrant up. Vagrant will create all virtual machines on your computer (run1-run10) and start them. To start only one (or few) VMs, pass the VM name(s) as parameters. Once the VM has booted up, login to the VM using SSH. $ vagrant up run1 $ vagrant ssh run1
5. Then inside ~/script $ cd script
6. Start the script run_allpuma.sh and start running $ bash –x run_allpuma.sh

2) Description of Files and Directories

1. In the /home/vagrant/

android-ndk-r10 #--> Android NDK

android-sdk-linux #--> Android SDK

lib #--> Libraries needed by tools

scripts #--> Scripts for experiments (invokes tools)

subjects #--> Open source android app benchmarks

tools #--> Android test input generation tools (include PUMA)

2. In the /vagrant(Shared folders)

Vagrantfile #--> Host machine directory (i.e., ~/vagrant/androtest) is mounted as /vagrant on the Vagrant box

3. /home/vagrant/script/

projects_34.txt #--> 34 App list for test

run_allpuma.sh #--> Parameter selection，t represents the app runtime, i represents waittime, j represents ui state, k represents search strategy. The meaning of a specific value representation is clearly explained in the script. For example, (0, 0, 0, 0) represents t=0(1hour), i=0(waitforidle), j=0(cosine), k=0(BFS).

run_puma.sh #-->puma运行脚本

3) Files Genearted

1. test.log – log generated by puma

2. tool.log -- log generated by the tool

3. tool.logcat -- logcat from the emulator, while tool was running (contains failure stack traces)

4. install.log -- installation log of the app on the device

5. icoverage -- log of intermediate coverage collected

6. coverage.em -- Emma coverage metadata

7. coverage.ec or coverage.es -- complete coverage file

8. coverage[1-11].ec -- snapshots of progressive coverage collected every 5 minutes.

4) Results Extraction

1. dataDeal.java –to generate coverage information by extracting line% from coverage.html;
2. findCrashByApp.java –to generate the number of failures by extracting Exception info from tool.logcat.
3. Usage method:

a. New a project named findCrash(for failure)/ dataDeal(for coverage), then new a class named findCrashByApp(for failure)/ dataDeal(for coverage). Then copy the corresponding .java file in the ~/tools/PUMA/

b. Parameter selection by changing the range of i, j, k; Turn the variable fileName to your local directory.

c. run the project and collect the result.

Contact Dr. Bo Jiang in case you have comments or problems with the work. [email protected], http://jiangbo.buaa.edu.cn