This directory is used by the bench subproject of the Scala sbt build.
It makes use of the sbt plugin for JMH.
Benchmarks are built with the bootstrap compiler ("starr") using the library built from the library project ("quick").
If you want to test compiler changes you need to bootstrap with the new compiler.
You'll then need to know the fully-qualified name of the benchmark runner class.
The benchmarking classes are organized under src/main/scala,
in the same package hierarchy as the classes that they test.
Assuming that we're benchmarking scala.collection.mutable.OpenHashMap,
the benchmark runner would likely be named scala.collection.mutable.OpenHashMapRunner.
Using this example, one would simply run
bench/jmh:runMain scala.collection.mutable.OpenHashMapRunner
in the Scala sbt build.
The JMH results can be found under ../../target/jmh-results/ (i.e. the main Scala build's target,
not the one that contains the benchmark class files). jmh-results gets deleted on an sbt bench/clean,
so you should copy these files out of target if you wish to preserve them.
The benchmarking classes use the same package hierarchy as the classes that they test in order to make it easy to expose, in package scope, members of the class under test, should that be necessary for benchmarking.
There are two types of classes in the source directory:
those suffixed Benchmark and those suffixed Runner.
The former are benchmarks that can be run directly using bench/jmh:run;
however, they are normally run from a corresponding class of the latter type,
which is run using bench/jmh:runMain (as described above).
This …Runner class is useful for setting appropriate JMH command options,
and for processing the JMH results into files that can be read by other tools, such as Gnuplot.
The benchmark.JmhRunner trait should be woven into any runner class, for the standard behavior that it provides.
This includes creating output files in a subdirectory of target/jmh-results
derived from the fully-qualified package name of the Runner class.
Adding these to the jmh:run or jmh:runMain command line may help if you're using the HotSpot (Oracle, OpenJDK) compiler.
They require prefixing with -jvmArgs.
See the Java documentation for more options.
Adding -XX:+PrintCompilation shows when Java methods are being compiled or deoptimized.
At the most basic level,
these messages will tell you whether the code that you're measuring is still being tuned,
so that you know whether you're running enough warm-up iterations.
See Kris Mok's notes to interpret the output in detail.
If you're not explicitly performing System.gc() calls outside of your benchmarking code,
you should add the JVM option -verbose:gc to understand the effect that GCs may be having on your tests.
These require the -XX:+UnlockDiagnosticVMOptions JVM option.
Add -XX:+PrintInlining.
If you're running OpenJDK or Oracle JVM,
you may need to install the disassembler library (hsdis-amd64.so for the amd64 architecture).
In Debian, this is available in
the libhsdis0-fcml package.
For an Oracle (or other compatible) JVM not set up by your distribution,
you may also need to copy or link the disassembler library
to the jre/lib/architecture directory inside your JVM installation directory.
The JITWatch project has hsdis build instructions. One way to obtain HSDIS is to use the binaries which are used in the Graal build.
To show the assembly code corresponding to the code generated by the JIT compiler for specific methods,
add -XX:CompileCommand=print,scala.collection.mutable.OpenHashMap::*,
for example, to show all of the methods in the scala.collection.mutable.OpenHashMap class.
To show it for all methods, add -XX:+PrintAssembly.
(This is usually excessive.)
JITWatch is useful to understand how the JVM has JIT compiled code.
If you install hsdis, as described above, machine code disassembly is also created.
You can generate the hotspot.log file for a benchmark run by adding the required JVM options
to JMH benchmark execution:
sbt:root> bench/jmh:run scala.collection.mutable.ArrayOpsBenchmark.insertInteger -psize=1000 -f1 -jvmArgs -XX:+UnlockDiagnosticVMOptions -jvmArgs -XX:+TraceClassLoading -jvmArgs -XX:+LogCompilation -jvmArgs -XX:LogFile=target/hotspot.log -jvmArgs -XX:+PrintAssembly
...
[info] Loaded disassembler from /Users/jz/.jabba/jdk/1.8.172/Contents/Home/jre/lib/hsdis-amd64.dylib
[info] Decoding compiled method 0x0000000113f60bd0:
[info] Code:
[info] [Disassembling for mach='i386:x86-64']
[info] [Entry Point]
[info] [Constants]
[info] # {method} {0x000000010ffa0000} 'hashCode' '()I' in 'java/lang/String'
[info] # [sp+0x40] (sp of caller)
[info] 0x0000000113f60d40: mov r10d,DWORD PTR [rsi+0x8]
[info] 0x0000000113f60d44: shl r10,0x3
...
[info] # Run complete. Total time: 00:00:30
[info] Benchmark (size) Mode Cnt Score Error Units
[info] ArrayOpsBenchmark.insertInteger 1000 avgt 10 188199.582 ± 5930.520 ns/op
JITWatch requires configuration of the class and source path. We generate that with a custom task in our build:
sbt> bench/jmh:jitwatchConfigFile
...
jmh
...
[info] ^-- UNRESOLVED DEPENDENCIES warnings above are normal, please ignore
[info] After cloning https://github.com/AdoptOpenJDK/jitwatch to $JITWATCH_HOME, compile and launch with:
[info] mvn -f $JITWATCH_HOME clean compile exec:java -Djitwatch.config.file=/Users/jz/code/scala/test/benchmarks/target/jitwatch-compile.properties -Djitwatch.logfile=/Users/jz/code/scala/test/benchmarks/target/hotspot.log
[info] Note: Add, for example, `-Djitwatch.focus.member="scala/collection/mutable/ArrayOpsBenchmark insertInteger (Lorg/openjdk/jmh/infra/Blackhole;)V"` to focus UI on a method of interest.
sbt> ^C
Follow instructions in the output above and start gleaning insights!
- OpenJDK advice on microbenchmarks
- Brian Goetz's "Java theory and practice" articles:
- Doug Lea's JSR 166 benchmarks
- "Measuring performance" of Scala parallel collections