Reproducible builds, composing different CBT version and various improvements

One large commit, because it is was hard to do these things in isolation or to separate them now.

CBT now knows how to load other versions of itself
 - Support for reproducible builds (!), by providing a CBT git URL and hash to tie build to
 - Support for composing builds using different CBT versions (!)
 - introduce (in compatibility/) Java interfaces all CBT versions need  to stay compatible with, so they can talk to each other. And put extension methods to these interfaces in cbt package object

Class loading
 - add some sanity checks for class loading
 - improve class loader invalidation to fix bugs
 - implement caching in Java land class loaders. In particular to prevent the system class loader to repeatedly generate ClassNotFound exceptions in each sink of the class loader DAG for non JDK classes (meaning major speed up for projects with many classes).
 - getting rid of transient class loader cache unifying into "persistent" one instead (which is still wrong as invalidation eventually needs to invalidate entire sub graphs of the class loading DAG, not single class loaders. Seems like we'll have to abandon the hashmap based approach and tie caching to dependency objects)

Other Caching
 - cache dependencies extracted from xml files, which was one major time killer, but invalidate cache when cbt changed (maven dependency user facing api needs simplification now!)
 - memorize last successful compile time in the file system rather than memory, to guard against unnecessary recompiling even across launches (or when using cbt direct)

Structural improvements
 - Factor out ClassLoaderCache on Java land into its own class.
 - Port MultiClassLoader to Java land, to better compose classloaders in NailgunLauncher.
 - Remove many global constants and variables (in object paths and in NailgunLauncher) and pass them through instead. Needed for composing of builds.
 - move more code from resolver into Lib for less entanglement with classes (needed to compatibility interfaces) and better re-usability
 - remove canBeCached. Everything can be cached now, but we need to be careful about correct invalidation.
 - remove build announcing produced jars. We can add if ever needed.
 - change callNullary to return exit code instead of Unit as preparation for next commit introducing "recursive"

ScalaTest
 - Makes ScalaTest support work (still a bit too inflexible, but mostly works well)

5 files changed, 87 insertions, 21 deletions

diff --git a/test/build/build.scala b/test/build/build.scala
index 29665a6..8989431 100644
--- a/test/build/build.scala
+++ b/test/build/build.scala
@@ -2,5 +2,5 @@ import cbt._
 import java.io.File
 import scala.collection.immutable.Seq
 class Build(context: cbt.Context) extends BasicBuild(context){
-  override def dependencies = Seq( CbtDependency() ) ++ super.dependencies 
+  override def dependencies = Seq( context.cbtDependency ) ++ super.dependencies 
 }
diff --git a/test/simple-fixed/Main.scala b/test/simple-fixed/Main.scala
new file mode 100644
index 0000000..1c423ca
--- /dev/null
+++ b/test/simple-fixed/Main.scala
@@ -0,0 +1,6 @@
+import ai.x.diff
+import org.eclipse.jgit.lib.Ref
+import com.spotify.missinglink.ArtifactLoader
+object Main extends App{
+  println(diff.DiffShow.diff("a","b"))
+}
diff --git a/test/simple-fixed/build/build.scala b/test/simple-fixed/build/build.scala
new file mode 100644
index 0000000..d088a40
--- /dev/null
+++ b/test/simple-fixed/build/build.scala
@@ -0,0 +1,34 @@
+import cbt._
+import cbt.extensions._ // FIXME: do not require this import
+import scala.collection.immutable.Seq
+import java.io.File
+
+// cbt:file:///Users/chris/code/cbt/#b65159f95421d9484f29327c11c0fa179eb7483f
+class Build(context: cbt.Context) extends BasicBuild(context){
+  override def dependencies = (
+    super.dependencies
+    ++
+    Seq(
+      GitDependency("https://github.com/xdotai/diff.git", "80a08bf45f7c4c3fd20c4bc6dbc9cae0072e3c0f"),
+      MavenResolver(context.cbtHasChanged,context.paths.mavenCache,MavenResolver.central).resolve(
+        ScalaDependency("com.typesafe.play", "play-json", "2.4.4"),
+        MavenDependency("joda-time", "joda-time", "2.9.2"),
+        // the below tests pom inheritance with dependencyManagement and variable substitution for pom properties
+        MavenDependency("org.eclipse.jgit", "org.eclipse.jgit", "4.2.0.201601211800-r"),
+        // the below tests pom inheritance with variable substitution for pom xml tag contents
+        MavenDependency("com.spotify", "missinglink-core", "0.1.1")
+      ),
+      MavenResolver(
+        context.cbtHasChanged,
+        context.paths.mavenCache,
+        MavenResolver.central,
+        MavenResolver.bintray("tpolecat"),
+        MavenResolver.sonatypeSnapshots
+      ).resolve(
+        "org.cvogt" %% "play-json-extensions" % "0.8.0",
+        "org.tpolecat" %% "tut-core" % "0.4.2",
+        "ai.x" %% "lens" % "1.0.0-SNAPSHOT"
+      )
+    )
+  )
+}
diff --git a/test/simple/build/build.scala b/test/simple/build/build.scala
index 5782c2d..f54c3dc 100644
--- a/test/simple/build/build.scala
+++ b/test/simple/build/build.scala
@@ -7,8 +7,9 @@ class Build(context: cbt.Context) extends BasicBuild(context){
     super.dependencies
     ++
     Seq(
-      GitDependency("https://github.com/xdotai/diff.git", "666bbbf4dbff6fadc81c011ade7b83e91d3f9256"),
-      MavenRepository.central.resolve(
+      GitDependency("https://github.com/xdotai/diff.git", "698717469b8dd86e8570b86354892be9c0654caf"),
+      // FIXME: make the below less verbose
+      MavenResolver(context.cbtHasChanged,context.paths.mavenCache,MavenResolver.central).resolve(
         ScalaDependency("com.typesafe.play", "play-json", "2.4.4"),
         MavenDependency("joda-time", "joda-time", "2.9.2"),
         // the below tests pom inheritance with dependencyManagement and variable substitution for pom properties
@@ -21,10 +22,12 @@ class Build(context: cbt.Context) extends BasicBuild(context){
         // TODO: put in a proper error message for version range not supported
         //MavenDependency("com.github.nikita-volkov", "sext", "0.2.4")
       ),
-      MavenRepository.combine(
-        MavenRepository.central,
-        MavenRepository.bintray("tpolecat"),
-        MavenRepository.sonatypeSnapshots
+      MavenResolver(
+        context.cbtHasChanged,
+        context.paths.mavenCache,
+        MavenResolver.central,
+        MavenResolver.bintray("tpolecat"),
+        MavenResolver.sonatypeSnapshots
       ).resolve(
         "org.cvogt" %% "play-json-extensions" % "0.8.0",
         "org.tpolecat" %% "tut-core" % "0.4.2",
diff --git a/test/test.scala b/test/test.scala
index 29e6afa..4cee1f1 100644
--- a/test/test.scala
+++ b/test/test.scala
@@ -1,14 +1,16 @@
 import cbt._
-import cbt.paths._
 import scala.collection.immutable.Seq
+import java.util.concurrent.ConcurrentHashMap
 import java.io.File
 
 // micro framework
 object Main{
   def main(_args: Array[String]): Unit = {
+    val start = System.currentTimeMillis
     val args = new Stage1ArgsParser(_args.toVector)
     implicit val logger: Logger = new Logger(args.enabledLoggers, System.currentTimeMillis)
     val lib = new Lib(logger)
+    val cbtHome = new File(System.getenv("CBT_HOME"))
     
     var successes = 0
     var failures = 0
@@ -71,11 +73,32 @@ object Main{
 
     logger.test( "Running tests " ++ _args.toList.toString )
 
+    val cache = cbtHome ++ "/cache"
+    val mavenCache = cache ++ "/maven"
+    val cbtHasChanged = true
+    val mavenCentral = MavenResolver(cbtHasChanged, mavenCache, MavenResolver.central)
+
     {
-      val noContext = Context(cbtHome ++ "/test/nothing", cbtHome, Seq(), logger, false, new ClassLoaderCache(logger))
-      val b = new Build(noContext){
+      val noContext = ContextImplementation(
+        cbtHome ++ "/test/nothing",
+        cbtHome,
+        Array(),
+        Array(),
+        start,
+        cbtHasChanged,
+        null,
+        null,
+        new ConcurrentHashMap[String,AnyRef],
+        new ConcurrentHashMap[AnyRef,ClassLoader],
+        cache,
+        cbtHome,
+        cbtHome ++ "/compatibilityTarget",
+        null
+      )
+
+      val b = new BasicBuild(noContext){
         override def dependencies = Seq(
-          MavenRepository.central.resolve(
+          mavenCentral.resolve(
             MavenDependency("net.incongru.watchservice","barbary-watchservice","1.0"),
             MavenDependency("net.incongru.watchservice","barbary-watchservice","1.0")
           )
@@ -87,34 +110,34 @@ object Main{
 
     // test that messed up artifacts crash with an assertion (which should tell the user what's up)
     assertException[AssertionError](){
-      MavenRepository.central.resolveOne( MavenDependency("com.jcraft", "jsch", " 0.1.53") ).classpath
+      mavenCentral.resolveOne( MavenDependency("com.jcraft", "jsch", " 0.1.53") ).classpath
     }
     assertException[AssertionError](){
-      MavenRepository.central.resolveOne( MavenDependency("com.jcraft", null, "0.1.53") ).classpath
+      mavenCentral.resolveOne( MavenDependency("com.jcraft", null, "0.1.53") ).classpath
     }
     assertException[AssertionError](){
-      MavenRepository.central.resolveOne( MavenDependency("com.jcraft", "", " 0.1.53") ).classpath
+      mavenCentral.resolveOne( MavenDependency("com.jcraft", "", " 0.1.53") ).classpath
     }
     assertException[AssertionError](){
-      MavenRepository.central.resolveOne( MavenDependency("com.jcraft%", "jsch", " 0.1.53") ).classpath
+      mavenCentral.resolveOne( MavenDependency("com.jcraft%", "jsch", " 0.1.53") ).classpath
     }
     assertException[AssertionError](){
-      MavenRepository.central.resolveOne( MavenDependency("", "jsch", " 0.1.53") ).classpath
+      mavenCentral.resolveOne( MavenDependency("", "jsch", " 0.1.53") ).classpath
     }
 
     (
-      MavenRepository.combine(
-        MavenRepository.central, MavenRepository.bintray("tpolecat")
+      MavenResolver(
+        cbtHasChanged, mavenCache, MavenResolver.central, MavenResolver.bintray("tpolecat")
       ).resolve(
         lib.ScalaDependency("org.tpolecat","tut-core","0.4.2", scalaMajorVersion="2.11")
       ).classpath.strings
       ++
-      MavenRepository.sonatype.resolve(
+      MavenResolver(cbtHasChanged, mavenCache,MavenResolver.sonatype).resolve(
         MavenDependency("org.cvogt","play-json-extensions_2.11","0.8.0")
       ).classpath.strings
       ++
-      MavenRepository.combine(
-        MavenRepository.central, MavenRepository.sonatypeSnapshots
+      MavenResolver(
+        cbtHasChanged, mavenCache, MavenResolver.central, MavenResolver.sonatypeSnapshots
       ).resolve(
         MavenDependency("ai.x","lens_2.11","1.0.0-SNAPSHOT")
       ).classpath.strings