Add JMH to the benchmark framework.

Add an example benchmark for OpenHashMap.

1 files changed, 65 insertions, 0 deletions

diff --git a/test/benchmarks/README.md b/test/benchmarks/README.md
new file mode 100644
index 0000000000..99b358dd99
--- /dev/null
+++ b/test/benchmarks/README.md
@@ -0,0 +1,65 @@
+# Scala library benchmarks
+
+This directory is a standalone SBT project
+that makes use of the [SBT plugin for JMH](https://github.com/ktoso/sbt-jmh),
+with the usual directory structure:
+source code for the benchmarks, which utilize [JMH](http://openjdk.java.net/projects/code-tools/jmh/),
+should be placed in `src/main/scala`.
+
+The benchmarks require first building Scala into `../../build/pack`.
+They can then be (built and) run from `sbt` with "`jmh:run`".
+"`jmh:run -h`" displays the usual JMH options available.
+
+## some useful HotSpot options
+Adding these to the `jmh:run` command line may help if you're using the HotSpot (Oracle, OpenJDK) compiler.
+They require prefixing with `-jvmArgs`.
+See [the Java documentation](http://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html) for more options. 
+
+### viewing JIT compilation events
+Adding `-XX:+PrintCompilation` shows when Java methods are being compiled or deoptimized.
+At the most basic level,
+these messages will tell you whether the code that you're measuring is still being tuned,
+so that you know whether you're running enough warm-up iterations.
+See [Kris Mok's notes](https://gist.github.com/rednaxelafx/1165804#file-notes-md) to interpret the output in detail.
+
+### consider GC events
+If you're not explicitly performing `System.gc()` calls outside of your benchmarking code,
+you should add the JVM option `-verbose:gc` to understand the effect that GCs may be having on your tests.
+
+### "diagnostic" options
+These require the `-XX:+UnlockDiagnosticVMOptions` JVM option.
+
+#### viewing inlining events
+Add `-XX:+PrintInlining`.
+
+#### viewing the disassembled code
+To show the assembly code corresponding to the code generated by the JIT compiler for specific methods,
+add `-XX:CompileCommand=print,scala.collection.mutable.OpenHashMap::*`,
+for example, to show all of the methods in the `scala.collection.mutable.OpenHashMap` class.
+If you're running OpenJDK, you may need to install the disassembler library (`hsdis-amd64.so` for the `amd64` architecture).
+In Debian, this is available in <a href="https://packages.debian.org/search?keywords=libhsdis0-fcml">the `libhsdis0-fcml` package</a>.
+
+To show it for _all_ methods, add `-XX:+PrintAssembly`.
+(This is usually excessive.)
+
+## useful reading
+* [OpenJDK advice on microbenchmarks](https://wiki.openjdk.java.net/display/HotSpot/MicroBenchmarks)
+* "[Measuring performance](http://docs.scala-lang.org/overviews/parallel-collections/performance.html)" of Scala parallel collections
+* Brian Goetz's "Java theory and practice" articles:
+  * "[Dynamic compilation and performance measurement](http://www.ibm.com/developerworks/java/library/j-jtp12214/)"
+  * "[Anatomy of a flawed benchmark](http://www.ibm.com/developerworks/java/library/j-jtp02225/)"
+
+## legacy frameworks
+
+An older version of the benchmarking framework is still present in this directory, in the following locations:
+
+<dl>
+<dt><code>bench</code></dt>
+<dd>A script to run the old benchmarks.</dd>
+<dt><code>source.list</code></dt>
+<dd>A temporary file used by <code>bench</code>.</dd>
+<dt><code>src/scala/</code></dt>
+<dd>The older benchmarks, including the previous framework.</dd>
+</dl>
+
+Another, older set of benchmarks is present in `../benchmarking/`.