MUBench comes as a Docker container that is controlled via the command line. Run ./mubench -h for details about the available commands and options.
We provide the same command-line interface for Linux/OSX via ./mubench and for Windows via ./mubench.bat.
Docker limits the computing resources available to a docker run. You can adjust this in the advanced preferences. Our recommendations are:
- CPUs: ≥2
- Memory: ≥8.0 GB
Remember that you may also have to provide more memory to the JVM for the detector run, for example, by passing --java-options Xmx8G to the pipeline invocation.
The easiest way to run experiments is to execute
./mubench publish findings <D> <E> -s <R> -u <RU> -p <RP>
Where <D> is the id of the detector, <E> is the number of the experiment to run, <R> is the URL of your review site, <RU> is the user name to access your review site as, and <RP> is the respective password.
Run ./mubench detect -h for a list of available detector ids.
Available experiments are:
- Provide detectors with example code of a correct usage, i.e., a usage pattern, to evaluate the recall of their detection strategy with respect to the misuses in the MUBench Dataset in isolation.
- Run detectors "in the wild", i.e., both their pattern mining and detection, to evaluate their precision in an end-user setting.
- Run detectors "in the wild", i.e., both their pattern mining and detection, to evaluate their recall with respect to the misuses in the MUBench Dataset.
MUBench runs the detectors on the projects/misuses specified in the MUBench Dataset. The first time a project is used in benchmarking, the repository containing that project is cloned (this may take a while). Subsequently, the existing clone is used, such that benchmarking runs offline. Before the first detector is run on a project, MUBench compiles the project (this may take a while). Subsequently, the compiled classes are reused. Then the detector is invoked, and finally the results are published to the review site.
You may run individual benchmark steps or select subsets of the entire dataset. See ./mubench -h for details.
You can specify defaults for command-line arguments by creating a ./default.config. The file follows the YAML format. Values for all command-line arguments that begin with -- can be speficied by using their respective full name as a key. To set command-line flags by default, set their value to True. For an example configuration, see the default.example.config. Argument values specified on the command line always take precedence over respective default values.