Reproducible builds, composing different CBT version and various improvements

One large commit, because it is was hard to do these things in isolation or to separate them now.

CBT now knows how to load other versions of itself
 - Support for reproducible builds (!), by providing a CBT git URL and hash to tie build to
 - Support for composing builds using different CBT versions (!)
 - introduce (in compatibility/) Java interfaces all CBT versions need  to stay compatible with, so they can talk to each other. And put extension methods to these interfaces in cbt package object

Class loading
 - add some sanity checks for class loading
 - improve class loader invalidation to fix bugs
 - implement caching in Java land class loaders. In particular to prevent the system class loader to repeatedly generate ClassNotFound exceptions in each sink of the class loader DAG for non JDK classes (meaning major speed up for projects with many classes).
 - getting rid of transient class loader cache unifying into "persistent" one instead (which is still wrong as invalidation eventually needs to invalidate entire sub graphs of the class loading DAG, not single class loaders. Seems like we'll have to abandon the hashmap based approach and tie caching to dependency objects)

Other Caching
 - cache dependencies extracted from xml files, which was one major time killer, but invalidate cache when cbt changed (maven dependency user facing api needs simplification now!)
 - memorize last successful compile time in the file system rather than memory, to guard against unnecessary recompiling even across launches (or when using cbt direct)

Structural improvements
 - Factor out ClassLoaderCache on Java land into its own class.
 - Port MultiClassLoader to Java land, to better compose classloaders in NailgunLauncher.
 - Remove many global constants and variables (in object paths and in NailgunLauncher) and pass them through instead. Needed for composing of builds.
 - move more code from resolver into Lib for less entanglement with classes (needed to compatibility interfaces) and better re-usability
 - remove canBeCached. Everything can be cached now, but we need to be careful about correct invalidation.
 - remove build announcing produced jars. We can add if ever needed.
 - change callNullary to return exit code instead of Unit as preparation for next commit introducing "recursive"

ScalaTest
 - Makes ScalaTest support work (still a bit too inflexible, but mostly works well)

10 files changed, 441 insertions, 244 deletions

diff --git a/stage1/ClassLoaderCache.scala b/stage1/ClassLoaderCache.scala
index 44b8d7d..e430ee1 100644
--- a/stage1/ClassLoaderCache.scala
+++ b/stage1/ClassLoaderCache.scala
@@ -4,15 +4,14 @@ import java.net._
 import java.util.concurrent.ConcurrentHashMap
 import collection.JavaConversions._
 
-class ClassLoaderCache(logger: Logger){
+case class ClassLoaderCache(
+  logger: Logger,
+  private[cbt] permanentKeys: ConcurrentHashMap[String,AnyRef],
+  private[cbt] permanentClassLoaders: ConcurrentHashMap[AnyRef,ClassLoader]
+){
   val persistent = new KeyLockedLazyCache(
-    NailgunLauncher.classLoaderCacheKeys.asInstanceOf[ConcurrentHashMap[String,AnyRef]],
-    NailgunLauncher.classLoaderCacheValues.asInstanceOf[ConcurrentHashMap[AnyRef,ClassLoader]],
-    Some(logger)
-  )
-  val transient = new KeyLockedLazyCache(
-    new ConcurrentHashMap[String,AnyRef],
-    new ConcurrentHashMap[AnyRef,ClassLoader],
+    permanentKeys,
+    permanentClassLoaders,
     Some(logger)
   )
   override def toString = (
@@ -20,10 +19,6 @@ class ClassLoaderCache(logger: Logger){
     ++
     persistent.keys.keySet.toVector.map(_.toString.split(":").mkString("\n")).sorted.mkString("\n\n","\n\n","\n\n")
     ++
-    "---------"
-    ++
-    transient.keys.keySet.toVector.map(_.toString.split(":").mkString("\n")).sorted.mkString("\n\n","\n\n^","\n\n")
-    ++
     ")"
   )
 }
diff --git a/stage1/ContextImplementation.scala b/stage1/ContextImplementation.scala
new file mode 100644
index 0000000..d8b1382
--- /dev/null
+++ b/stage1/ContextImplementation.scala
@@ -0,0 +1,22 @@
+package cbt
+import java.io._
+import java.util.concurrent.ConcurrentHashMap
+import scala.collection.immutable.Seq
+import java.lang._
+
+case class ContextImplementation(
+  projectDirectory: File,
+  cwd: File,
+  argsArray: Array[String],
+  enabledLoggersArray: Array[String],
+  startCompat: Long,
+  cbtHasChangedCompat: Boolean,
+  versionOrNull: String,
+  scalaVersionOrNull: String,
+  permanentKeys: ConcurrentHashMap[String,AnyRef],
+  permanentClassLoaders: ConcurrentHashMap[AnyRef,ClassLoader],
+  cache: File,
+  cbtHome: File,
+  compatibilityTarget: File,
+  parentBuildOrNull: BuildInterface
+) extends Context
\ No newline at end of file
diff --git a/stage1/KeyLockedLazyCache.scala b/stage1/KeyLockedLazyCache.scala
index aca5f74..4eff5b2 100644
--- a/stage1/KeyLockedLazyCache.scala
+++ b/stage1/KeyLockedLazyCache.scala
@@ -47,8 +47,16 @@ final private[cbt] class KeyLockedLazyCache[Key <: AnyRef,Value <: AnyRef](
   def remove( key: Key ) = keys.synchronized{    
     assert(keys containsKey key)
     val lockableKey = keys get key
-    keys.remove( key )
-    assert(values containsKey lockableKey)
-    values.remove( lockableKey )
+    lockableKey.synchronized{
+      if(values containsKey lockableKey){
+        // this is so values in the process of being replaced (which mean they have a key but no value)
+        // are not being removed
+        keys.remove( key )
+        values.remove( lockableKey )  
+      }
+    }
+  }
+  def containsKey( key: Key ) = keys.synchronized{
+    keys containsKey key
   }
 }
diff --git a/stage1/MavenRepository.scala b/stage1/MavenRepository.scala
index bfd52a7..aa31cb8 100644
--- a/stage1/MavenRepository.scala
+++ b/stage1/MavenRepository.scala
@@ -1,22 +1,19 @@
 package cbt
 import scala.collection.immutable.Seq
+import java.io._
 import java.net._
-case class MavenRepository(url: URL){
-    def resolve( dependencies: MavenDependency* )(implicit logger: Logger): BoundMavenDependencies
-      = new BoundMavenDependencies( Seq(url), dependencies.to )
-    def resolveOne( dependency: MavenDependency )(implicit logger: Logger): BoundMavenDependency
-      = BoundMavenDependency( dependency, Seq(url) )
+case class MavenResolver( cbtHasChanged: Boolean, mavenCache: File, urls: URL* ){
+  def resolve( dependencies: MavenDependency* )(implicit logger: Logger): BoundMavenDependencies
+    = new BoundMavenDependencies( cbtHasChanged, mavenCache, urls.to, dependencies.to )
+  def resolveOne( dependency: MavenDependency )(implicit logger: Logger): BoundMavenDependency
+    = BoundMavenDependency( cbtHasChanged, mavenCache, dependency, urls.to )
 }
 
-object MavenRepository{
-  case class combine(repositories: MavenRepository*){
-    def resolve( dependencies: MavenDependency* )(implicit logger: Logger): BoundMavenDependencies
-      = new BoundMavenDependencies( repositories.map(_.url).to, dependencies.to )
-  }
-  def central = MavenRepository(new URL(NailgunLauncher.MAVEN_URL))
-  def jcenter = MavenRepository(new URL("https://jcenter.bintray.com/releases"))
-  def bintray(owner: String) = MavenRepository(new URL(s"https://dl.bintray.com/$owner/maven"))
+object MavenResolver{
+  def central = new URL("https://repo1.maven.org/maven2")
+  def jcenter = new URL("https://jcenter.bintray.com/releases")
+  def bintray(owner: String) = new URL(s"https://dl.bintray.com/$owner/maven")
   private val sonatypeBase  = new URL("https://oss.sonatype.org/content/repositories/")
-  def sonatype          = MavenRepository(sonatypeBase ++ "releases")
-  def sonatypeSnapshots = MavenRepository(sonatypeBase ++ "snapshots")
+  def sonatype          = sonatypeBase ++ "releases"
+  def sonatypeSnapshots = sonatypeBase ++ "snapshots"
 }
diff --git a/stage1/MultiClassLoader.scala b/stage1/MultiClassLoader.scala
index 74e65aa..3e3ba26 100644
--- a/stage1/MultiClassLoader.scala
+++ b/stage1/MultiClassLoader.scala
@@ -3,7 +3,7 @@ import java.net._
 import scala.collection.immutable.Seq
 
 // do not make this a case class, required object identity equality
-class MultiClassLoader(parents: Seq[ClassLoader])(implicit val logger: Logger) extends ClassLoader with CachingClassLoader{
+class MultiClassLoader(final val parents: Seq[ClassLoader])(implicit val logger: Logger) extends ClassLoader(null) with CachingClassLoader{
   override def findClass(name: String) = {
     parents.find( parent =>
       try{
diff --git a/stage1/Stage1.scala b/stage1/Stage1.scala
index 77b88a2..bf2c272 100644
--- a/stage1/Stage1.scala
+++ b/stage1/Stage1.scala
@@ -1,13 +1,13 @@
 package cbt
 
 import java.io._
+import java.util.concurrent.ConcurrentHashMap
 
 import scala.collection.immutable.Seq
 import scala.collection.JavaConverters._
 
-import paths._
-
-final case class Stage1ArgsParser(_args: Seq[String]) {
+final case class Stage1ArgsParser(__args: Seq[String]) {
+  val _args = __args.drop(1)
   /**
    * Raw parameters including their `-D` flag.
   **/
@@ -41,60 +41,149 @@ case class Stage2Args(
   cwd: File,
   args: Seq[String],
   cbtHasChanged: Boolean,
-  logger: Logger,
-  classLoaderCache: ClassLoaderCache
-)
-
+  classLoaderCache: ClassLoaderCache,
+  cache: File,
+  cbtHome: File
+){
+  val ClassLoaderCache(
+    logger,
+    permanentKeys,
+    permanentClassLoaders
+  ) = classLoaderCache  
+}
 object Stage1{
   protected def newerThan( a: File, b: File ) ={
     a.lastModified > b.lastModified
   }
 
-  def run(_args: Array[String], classLoader: ClassLoader, _cbtChanged: java.lang.Boolean, start: java.lang.Long): Int = {
-    val args = Stage1ArgsParser(_args.toVector)
-    val logger = new Logger(args.enabledLoggers, start)
-    logger.stage1(s"Stage1 start")
+  def getBuild( _context: java.lang.Object, _cbtChanged: java.lang.Boolean ) = {
+    val context = _context.asInstanceOf[Context]
+    val logger = new Logger( context.enabledLoggers, context.start )
+    val (changed, classLoader) = buildStage2(
+      context.compatibilityTarget,
+      ClassLoaderCache(
+        logger,
+        context.permanentKeys,
+        context.permanentClassLoaders
+      ),
+      _cbtChanged,
+      context.cbtHome,
+      context.cache
+    )
+
+    classLoader
+      .loadClass("cbt.Stage2")
+      .getMethod( "getBuild", classOf[java.lang.Object], classOf[java.lang.Boolean] )
+      .invoke(null, context, (_cbtChanged || changed): java.lang.Boolean)
+  }
+
+  def buildStage2(
+    _compatibilityTarget: File, classLoaderCache: ClassLoaderCache, _cbtChanged: Boolean, cbtHome: File, cache: File
+  ): (Boolean, ClassLoader) = {
+    import classLoaderCache.logger
 
     val lib = new Stage1Lib(logger)
     import lib._
-    val classLoaderCache = new ClassLoaderCache(logger)
+    val paths = Paths(cbtHome, cache)
+    import paths._
+
+    val stage2sourceFiles = stage2.listFiles.toVector.filter(_.isFile).filter(_.toString.endsWith(".scala"))
+    val cbtHasChanged = _cbtChanged || lib.needsUpdate(stage2sourceFiles, stage2StatusFile)
+
+    val cls = this.getClass.getClassLoader.loadClass("cbt.NailgunLauncher")
+    
+    val cbtDependency = CbtDependency(cbtHasChanged, mavenCache, nailgunTarget, stage1Target, stage2Target, _compatibilityTarget)
 
-    val sourceFiles = stage2.listFiles.toVector.filter(_.isFile).filter(_.toString.endsWith(".scala"))
-    val cbtHasChanged = _cbtChanged || lib.needsUpdate(sourceFiles, stage2StatusFile)
     logger.stage1("Compiling stage2 if necessary")
     compile(
       cbtHasChanged,
-      sourceFiles, stage2Target, stage2StatusFile,
-      CbtDependency().dependencyClasspath,
+      cbtHasChanged,
+      stage2sourceFiles, stage2Target, stage2StatusFile,
+      cbtDependency.dependencyClasspath,
+      mavenCache,
       Seq("-deprecation"), classLoaderCache,
       zincVersion = "0.3.9", scalaVersion = constants.scalaVersion
     )
 
     logger.stage1(s"calling CbtDependency.classLoader")
-    if(cbtHasChanged) {
-      NailgunLauncher.stage2classLoader = CbtDependency().classLoader(classLoaderCache)
-    }else{
-      classLoaderCache.transient.get( CbtDependency().classpath.string, NailgunLauncher.stage2classLoader )
+    if( cbtHasChanged && classLoaderCache.persistent.containsKey( cbtDependency.classpath.string ) ) {
+      classLoaderCache.persistent.remove( cbtDependency.classpath.string )
     }
 
+    val stage2ClassLoader = classLoaderCache.persistent.get(
+      cbtDependency.classpath.string,
+      cbtDependency.classLoader(classLoaderCache)
+    )
+
+    {
+      // a few classloader sanity checks
+      val compatibilityClassLoader =
+        CompatibilityDependency(cbtHasChanged, compatibilityTarget)
+          .classLoader(classLoaderCache)
+      assert(
+        classOf[BuildInterface].getClassLoader == compatibilityClassLoader,
+        classOf[BuildInterface].getClassLoader.toString ++ "\n\nis not the same as\n\n" ++ compatibilityClassLoader.toString
+      )
+      //-------------
+      val stage1Dependency = Stage1Dependency(cbtHasChanged, mavenCache, nailgunTarget, stage1Target, compatibilityTarget)
+      val stage1ClassLoader =
+        stage1Dependency.classLoader(classLoaderCache)
+      assert(
+        classOf[Stage1Dependency].getClassLoader == stage1ClassLoader,
+        classOf[Stage1Dependency].getClassLoader.toString ++ "\n\nis not the same as\n\n" ++ stage1ClassLoader.toString
+      )
+      //-------------
+      assert(
+        Stage0Lib.get(stage2ClassLoader.getParent,"parents").asInstanceOf[Seq[ClassLoader]].contains(stage1ClassLoader),
+        stage1ClassLoader.toString ++ "\n\nis not contained in parents of\n\n" ++ stage2ClassLoader.toString
+      )
+    }
+
+    ( cbtHasChanged, stage2ClassLoader )
+  }
+
+  def run(
+    _args: Array[String],
+    cache: File,
+    cbtHome: File,
+    _cbtChanged: java.lang.Boolean,
+    start: java.lang.Long,
+    classLoaderCacheKeys: ConcurrentHashMap[String,AnyRef],
+    classLoaderCacheValues: ConcurrentHashMap[AnyRef,ClassLoader]
+  ): Int = {
+    val args = Stage1ArgsParser(_args.toVector)
+    val logger = new Logger(args.enabledLoggers, start)
+    logger.stage1(s"Stage1 start")
+
+    val classLoaderCache = ClassLoaderCache(
+      logger,
+      classLoaderCacheKeys,
+      classLoaderCacheValues
+    )
+    
+
+    val (cbtHasChanged, classLoader) = buildStage2( Paths(cbtHome, cache).compatibilityTarget, classLoaderCache, _cbtChanged, cbtHome, cache )
+
+    val stage2Args = Stage2Args(
+      new File( args.args(0) ),
+      args.args.drop(1).toVector,
+      // launcher changes cause entire nailgun restart, so no need for them here
+      cbtHasChanged = cbtHasChanged,
+      classLoaderCache = classLoaderCache,
+      cache,
+      cbtHome
+    )
+
     logger.stage1(s"Run Stage2")
-    val cl = NailgunLauncher.stage2classLoader
     val exitCode = (
-      cl
+      classLoader
       .loadClass(
         if(args.admin) "cbt.AdminStage2" else "cbt.Stage2"
       )
       .getMethod( "run", classOf[Stage2Args] )
       .invoke(
         null,
-        Stage2Args(
-          new File( args.args(0) ),
-          args.args.drop(1).toVector,
-          // launcher changes cause entire nailgun restart, so no need for them here
-          cbtHasChanged = cbtHasChanged,
-          logger,
-          classLoaderCache
-        )
+        stage2Args
       ) match {
         case code: ExitCode => code
         case _ => ExitCode.Success
diff --git a/stage1/Stage1Lib.scala b/stage1/Stage1Lib.scala
index f484bbf..5fd19a8 100644
--- a/stage1/Stage1Lib.scala
+++ b/stage1/Stage1Lib.scala
@@ -1,15 +1,15 @@
 package cbt
 
-import cbt.paths._
-
 import java.io._
 import java.lang.reflect.InvocationTargetException
 import java.net._
+import java.nio.charset.StandardCharsets
 import java.nio.file._
 import java.nio.file.attribute.FileTime
 import javax.tools._
 import java.security._
-import java.util._
+import java.util.{Set=>_,Map=>_,_}
+import java.util.concurrent.ConcurrentHashMap
 import javax.xml.bind.annotation.adapters.HexBinaryAdapter
 
 import scala.collection.immutable.Seq
@@ -31,19 +31,8 @@ object CatchTrappedExitCode{
   }
 }
 
-case class Context(
-  projectDirectory: File,
-  cwd: File,
-  args: Seq[String],
-  logger: Logger,
-  cbtHasChanged: Boolean,
-  classLoaderCache: ClassLoaderCache,
-  version: Option[String] = None,
-  scalaVersion: Option[String] = None
-)
-
 class BaseLib{
-  def realpath(name: File) = new File(Paths.get(name.getAbsolutePath).normalize.toString)
+  def realpath(name: File) = new File(java.nio.file.Paths.get(name.getAbsolutePath).normalize.toString)
 }
 
 class Stage1Lib( val logger: Logger ) extends BaseLib{
@@ -140,11 +129,13 @@ class Stage1Lib( val logger: Logger ) extends BaseLib{
   }
 
   def compile(
+    cbtHasChanged: Boolean,
     needsRecompile: Boolean,
     files: Seq[File],
     compileTarget: File,
     statusFile: File,
     classpath: ClassPath,
+    mavenCache: File,
     scalacOptions: Seq[String] = Seq(),
     classLoaderCache: ClassLoaderCache,
     zincVersion: String,
@@ -159,7 +150,7 @@ class Stage1Lib( val logger: Logger ) extends BaseLib{
       None
     }else{
       if( needsRecompile ){
-        import MavenRepository.central
+        val central = MavenResolver(cbtHasChanged, mavenCache,MavenResolver.central)
         val zinc = central.resolveOne(MavenDependency("com.typesafe.zinc","zinc", zincVersion))
         val zincDeps = zinc.transitiveDependencies
         
@@ -167,8 +158,7 @@ class Stage1Lib( val logger: Logger ) extends BaseLib{
           zincDeps
             .collect{ case d @
               BoundMavenDependency(
-                MavenDependency( "com.typesafe.sbt", "sbt-interface", _, Classifier.none),
-                _
+                _, _, MavenDependency( "com.typesafe.sbt", "sbt-interface", _, Classifier.none), _
               ) => d
             }
             .headOption
@@ -179,8 +169,7 @@ class Stage1Lib( val logger: Logger ) extends BaseLib{
           zincDeps
             .collect{ case d @
               BoundMavenDependency(
-                MavenDependency( "com.typesafe.sbt", "compiler-interface", _, Classifier.sources),
-                _
+                _, _, MavenDependency( "com.typesafe.sbt", "compiler-interface", _, Classifier.sources), _
               ) => d
             }
             .headOption
@@ -207,6 +196,7 @@ class Stage1Lib( val logger: Logger ) extends BaseLib{
 
         val code = 
           try{
+            System.err.println("Compiling to " ++ compileTarget.toString)
             redirectOutToErr{
               lib.runMain(
                 _class,
@@ -279,4 +269,93 @@ ${files.sorted.mkString(" \\\n")}
     MavenDependency(
       groupId, artifactId ++ "_" ++ scalaMajorVersion, version, classifier
     )
+
+  def cacheOnDisk[T]
+    ( cbtHasChanged: Boolean, cacheFile: File )
+    ( deserialize: String => T )
+    ( serialize: T => String )
+    ( compute: => Seq[T] ) = {
+    if(!cbtHasChanged && cacheFile.exists){
+      import collection.JavaConversions._
+      Files
+        .readAllLines( cacheFile.toPath, StandardCharsets.UTF_8 )
+        .toStream
+        .map(deserialize)
+    } else {
+      val result = compute
+      val string = result.map(serialize).mkString("\n")
+      Files.write(cacheFile.toPath, string.getBytes)
+      result
+    }
+  }
+
+  def dependencyTreeRecursion(root: Dependency, indent: Int = 0): String = (
+    ( " " * indent )
+    ++ (if(root.needsUpdate) red(root.show) else root.show)
+    ++ root.dependencies.map( d =>
+      "\n" ++ dependencyTreeRecursion(d,indent + 1)
+    ).mkString
+  )
+
+  def transitiveDependencies(dependency: Dependency): Seq[Dependency] = {
+    def linearize(deps: Seq[Dependency]): Seq[Dependency] = {
+      // Order is important here in order to generate the correct lineraized dependency order for EarlyDependencies
+      // (and maybe this as well in case we want to get rid of MultiClassLoader)
+      try{
+        if(deps.isEmpty) deps else ( deps ++ linearize(deps.flatMap(_.dependencies)) )
+      } catch{
+        case e: Exception => throw new Exception(dependency.show, e)
+      }
+    }
+
+    // FIXME: this is probably wrong too eager.
+    // We should consider replacing versions during traversals already
+    // not just replace after traversals, because that could mean we
+    // pulled down dependencies current versions don't even rely
+    // on anymore.
+
+    val deps: Seq[Dependency] = linearize(dependency.dependencies).reverse.distinct.reverse
+    val hasInfo: Seq[Dependency with ArtifactInfo] = deps.collect{ case d:Dependency with ArtifactInfo => d }
+    val noInfo: Seq[Dependency]  = deps.filter{
+      case _:Dependency with ArtifactInfo => false
+      case _ => true
+    }
+    noInfo ++ BoundMavenDependency.updateOutdated( hasInfo ).reverse.distinct
+  }
+
+
+  def actual(current: Dependency, latest: Map[(String,String),Dependency]) = current match {
+    case d: ArtifactInfo => latest((d.groupId,d.artifactId))
+    case d => d
+  }
+
+  def classLoaderRecursion( dependency: Dependency, latest: Map[(String,String),Dependency], cache: ClassLoaderCache ): ClassLoader = {
+    val d = dependency
+    val dependencies = dependency.dependencies
+    def dependencyClassLoader( latest: Map[(String,String),Dependency], cache: ClassLoaderCache ): ClassLoader = {
+      if( dependency.dependencies.isEmpty ){
+        // wrap for caching
+        new cbt.URLClassLoader( ClassPath(Seq()), ClassLoader.getSystemClassLoader().getParent() )
+      } else if( dependencies.size == 1 ){
+        classLoaderRecursion( dependencies.head, latest, cache )
+      } else{
+        val cp = d.dependencyClasspath.string
+        if( dependencies.exists(_.needsUpdate) && cache.persistent.containsKey(cp) ){
+          cache.persistent.remove(cp)
+        }
+        cache.persistent.get(
+          cp,
+          new MultiClassLoader(
+            dependencies.map( classLoaderRecursion(_, latest, cache) )
+          )
+        )
+      }
+    }
+
+    val a = actual( dependency, latest )
+    cache.persistent.get(
+      a.classpath.string,
+      new cbt.URLClassLoader( a.exportedClasspath, dependencyClassLoader(latest, cache) )
+    )
+  }
 }
diff --git a/stage1/cbt.scala b/stage1/cbt.scala
index 7b8b632..4594135 100644
--- a/stage1/cbt.scala
+++ b/stage1/cbt.scala
@@ -3,6 +3,7 @@ import java.io._
 import java.nio.file._
 import java.net._
 import scala.collection.immutable.Seq
+import java.util.concurrent.ConcurrentHashMap
 
 object `package`{
   private val lib = new BaseLib
@@ -20,4 +21,71 @@ object `package`{
     def ++( s: String ): URL = new URL( url.toString ++ s )
     def string = url.toString
   }
+  implicit class BuildInterfaceExtensions(build: BuildInterface){
+    import build._
+    def triggerLoopFiles: Seq[File] = triggerLoopFilesArray.to
+    def crossScalaVersions: Seq[String] = crossScalaVersionsArray.to
+  }
+  implicit class ArtifactInfoExtensions(subject: ArtifactInfo){
+    import subject._
+    def str = s"$groupId:$artifactId:$version"
+    def show = this.getClass.getSimpleName ++ s"($str)"
+  }
+  implicit class DependencyExtensions(subject: Dependency){
+    import subject._
+    def dependencyClasspath: ClassPath = ClassPath(dependencyClasspathArray.to)
+    def exportedClasspath: ClassPath = ClassPath(exportedClasspathArray.to)
+    def classpath = exportedClasspath ++ dependencyClasspath
+    def dependencies: Seq[Dependency] = dependenciesArray.to
+    def needsUpdate: Boolean = needsUpdateCompat
+  }
+  implicit class ContextExtensions(subject: Context){
+    import subject._
+    val paths = Paths(cbtHome, cache)
+    implicit def logger: Logger = new Logger(enabledLoggers, start)
+    def classLoaderCache: ClassLoaderCache = new ClassLoaderCache(
+      logger,
+      permanentKeys,
+      permanentClassLoaders
+    )
+    def cbtDependency = {
+      import paths._
+      CbtDependency(cbtHasChanged, mavenCache, nailgunTarget, stage1Target, stage2Target, compatibilityTarget)
+    }
+    def args: Seq[String] = argsArray.to
+    def enabledLoggers: Set[String] = enabledLoggersArray.to
+    def scalaVersion = Option(scalaVersionOrNull)
+    def version = Option(versionOrNull)
+    def parentBuild = Option(parentBuildOrNull)
+    def start: scala.Long = startCompat
+    def cbtHasChanged: scala.Boolean = cbtHasChangedCompat
+
+    def copy(
+      projectDirectory: File = projectDirectory,
+      args: Seq[String] = args,
+      enabledLoggers: Set[String] = enabledLoggers,
+      cbtHasChanged: Boolean = cbtHasChanged,
+      version: Option[String] = version,
+      scalaVersion: Option[String] = scalaVersion,
+      cache: File = cache,
+      cbtHome: File = cbtHome,
+      parentBuild: Option[BuildInterface] = None
+    ): Context = ContextImplementation(
+      projectDirectory,
+      cwd,
+      args.to,
+      enabledLoggers.to,
+      startCompat,
+      cbtHasChangedCompat,
+      version.getOrElse(null),
+      scalaVersion.getOrElse(null),
+      permanentKeys,
+      permanentClassLoaders,
+      cache,
+      cbtHome,
+      compatibilityTarget,
+      parentBuild.getOrElse(null)
+    )
+  }
 }
+
diff --git a/stage1/paths.scala b/stage1/paths.scala
index f27e538..c16c614 100644
--- a/stage1/paths.scala
+++ b/stage1/paths.scala
@@ -1,17 +1,17 @@
 package cbt
 import java.io._
-object paths{  
-  val cbtHome: File = new File(Option(System.getenv("CBT_HOME")).get)
-  val mavenCache: File = cbtHome ++ "/cache/maven"
+case class Paths(private val cbtHome: File, private val cache: File){
   val userHome: File = new File(Option(System.getProperty("user.home")).get)
-  val bootstrapScala: File = cbtHome ++ "/bootstrap_scala"
-  val nailgun: File = new File(Option(System.getenv("NAILGUN")).get)
-  val stage1: File = new File(NailgunLauncher.STAGE1)
+  val nailgun: File = cbtHome ++ "/nailgun_launcher"
+  val stage1: File = cbtHome ++ "/stage1"
   val stage2: File = cbtHome ++ "/stage2"
-  private val target = Option(System.getenv("TARGET")).get.stripSuffix("/")
+  val mavenCache: File = cache ++ "/maven"
+  private val target = NailgunLauncher.TARGET.stripSuffix("/")
   val stage1Target: File = stage1 ++ ("/" ++ target)
   val stage2Target: File = stage2 ++ ("/" ++ target)
   val stage2StatusFile: File = stage2Target ++ ".last-success"
+  val compatibility: File = cbtHome ++ "/compatibility"
+  val compatibilityTarget: File = compatibility ++ ("/" ++ target)
+  val compatibilityStatusFile: File = compatibilityTarget ++ ".last-success"
   val nailgunTarget: File = nailgun ++ ("/" ++ target)
-  val sonatypeLogin: File = cbtHome ++ "/sonatype.login"
 }
diff --git a/stage1/resolver.scala b/stage1/resolver.scala
index ad6df23..f979247 100644
--- a/stage1/resolver.scala
+++ b/stage1/resolver.scala
@@ -5,31 +5,24 @@ import java.net._
 import java.io._
 import scala.collection.immutable.Seq
 import scala.xml._
-import paths._
 import scala.concurrent._
 import scala.concurrent.duration._
 
-trait ArtifactInfo extends Dependency{
-  def artifactId: String
-  def groupId: String
-  def version: String
-
-  protected def str = s"$groupId:$artifactId:$version"
-  override def show = super.show ++ s"($str)"
-}
-abstract class Dependency{
-  implicit def logger: Logger
+abstract class DependencyImplementation extends Dependency{
+  implicit protected def logger: Logger
   protected def lib = new Stage1Lib(logger)
 
   def needsUpdate: Boolean
   //def cacheClassLoader: Boolean = false
   private[cbt] def targetClasspath: ClassPath
+  def dependencyClasspathArray: Array[File] = dependencyClasspath.files.toArray
+  def exportedClasspathArray: Array[File] = exportedClasspath.files.toArray
   def exportedClasspath: ClassPath
-  def exportedJars: Seq[File]
-  def jars: Seq[File] = exportedJars ++ dependencyJars
+  def dependenciesArray: Array[Dependency] = dependencies.to
 
-  def canBeCached: Boolean
+  def needsUpdateCompat: java.lang.Boolean = needsUpdate
 
+  /*
   //private type BuildCache = KeyLockedLazyCache[Dependency, Future[ClassPath]]
   def exportClasspathConcurrently: ClassPath = {
     // FIXME: this should separate a blocking and a non-blocking EC
@@ -41,7 +34,7 @@ abstract class Dependency{
           .groupBy( d => (d.groupId,d.artifactId) )
           .mapValues( _.head )
         //, new BuildCache
-      ),
+      ), // FIXME
       Duration.Inf
     )
   }
@@ -52,9 +45,9 @@ abstract class Dependency{
   The implementation of this method is untested and likely buggy
   at this stage.
   */
-  private def exportClasspathConcurrently(
-    latest: Map[(String, String),ArtifactInfo]//, cache: BuildCache
-  )( implicit ec: ExecutionContext ): Future[ClassPath] = {
+  def exportClasspathConcurrently(
+    latest: Map[(String, String),Dependency with ArtifactInfo]//, cache: BuildCache
+  )( implicit ec: ExecutionContext ): Future[AnyRef] = {
     Future.sequence( // trigger compilation / download of all dependencies first
       this.dependencies.map{
         d =>
@@ -65,7 +58,7 @@ abstract class Dependency{
           }
           // // trigger compilation if not already triggered
           // cache.get( l, l.exportClasspathConcurrently( latest, cache ) )
-          l.exportClasspathConcurrently( latest )
+          l.exportClasspathConcurrently( latest ) // FIXME
       }
     ).map(
       // merge dependency classpaths into one
@@ -76,168 +69,102 @@ abstract class Dependency{
       exportedClasspath
     )
   }
+  */
 
-  protected def actual(current: Dependency, latest: Map[(String,String),Dependency]) = current match {
-    case d: ArtifactInfo => latest((d.groupId,d.artifactId))
-    case d => d
-  }
-  private def dependencyClassLoader( latest: Map[(String,String),Dependency], cache: ClassLoaderCache ): ClassLoader = {
-    if( dependencies.isEmpty ){
-      ClassLoader.getSystemClassLoader
-    } else if( dependencies.size == 1 ){
-      dependencies.head.classLoaderRecursion( latest, cache )
-    } else if( dependencies.forall(_.canBeCached) ){
-      assert(transitiveDependencies.forall(_.canBeCached))
-      cache.persistent.get(
-        dependencyClasspath.string,
-        new MultiClassLoader(
-          dependencies.map( _.classLoaderRecursion(latest, cache) )
-        )
-      )
-    } else {
-      cache.transient.get(
-        dependencyClasspath.string,
-        new MultiClassLoader(
-          dependencies.map( _.classLoaderRecursion(latest, cache) )
-        )
-      )
-    }
-  }
-  protected def classLoaderRecursion( latest: Map[(String,String),Dependency], cache: ClassLoaderCache ): ClassLoader = {
-    val a = actual( this, latest )
-    (
-      if( canBeCached ){
-        cache.persistent
-      } else {
-        cache.transient
-      }
-    ).get(
-      a.classpath.string,
-      new cbt.URLClassLoader( a.exportedClasspath, dependencyClassLoader(latest, cache) )
-    )
-  }
-  def classLoader( cache: ClassLoaderCache  ): ClassLoader = {
+  def classLoader( cache: ClassLoaderCache ): ClassLoader = {
+    /*
     if( concurrencyEnabled ){
       // trigger concurrent building / downloading dependencies
       exportClasspathConcurrently
     }
-    classLoaderRecursion(
+    */
+    lib.classLoaderRecursion(
+      this,
       (this +: transitiveDependencies).collect{
         case d: ArtifactInfo => d 
       }.groupBy(
         d => (d.groupId,d.artifactId)
       ).mapValues(_.head),
-      cache
+      cache // FIXME
     )
   }
   // FIXME: these probably need to update outdated as well
   def classpath           : ClassPath = exportedClasspath ++ dependencyClasspath
-  def dependencyJars      : Seq[File] = transitiveDependencies.flatMap(_.jars)
-  def dependencyClasspath : ClassPath = ClassPath.flatten( transitiveDependencies.map(_.exportedClasspath) )
+  def dependencyClasspath : ClassPath = ClassPath(
+    transitiveDependencies
+    .flatMap(_.exportedClasspath.files)
+    .distinct // <- currently needed here to handle diamond dependencies on builds (duplicate in classpath)
+  )
   def dependencies: Seq[Dependency]
 
-  private def linearize(deps: Seq[Dependency]): Seq[Dependency] =
-    // Order is important here in order to generate the correct lineraized dependency order for EarlyDependencies
-    // (and maybe this as well in case we want to get rid of MultiClassLoader)
-    if(deps.isEmpty) deps else ( deps ++ linearize(deps.flatMap(_.dependencies)) )
-
   private object transitiveDependenciesCache extends Cache[Seq[Dependency]]
   /** return dependencies in order of linearized dependence. this is a bit tricky. */
   def transitiveDependencies: Seq[Dependency] = transitiveDependenciesCache{
-    // FIXME: this is probably wrong too eager.
-    // We should consider replacing versions during traversals already
-    // not just replace after traversals, because that could mean we
-    // pulled down dependencies current versions don't even rely
-    // on anymore.
-
-    val deps: Seq[Dependency] = linearize(dependencies).reverse.distinct.reverse
-    val hasInfo: Seq[Dependency with ArtifactInfo] = deps.collect{ case d:Dependency with ArtifactInfo => d }
-    val noInfo: Seq[Dependency]  = deps.filter{
-      case _:Dependency with ArtifactInfo => false
-      case _ => true
-    }
-    noInfo ++ BoundMavenDependency.updateOutdated( hasInfo ).reverse.distinct
+    lib.transitiveDependencies(this)
   }
 
   override def show: String = this.getClass.getSimpleName
   // ========== debug ==========
-  def dependencyTree: String = dependencyTreeRecursion()
-  private def dependencyTreeRecursion(indent: Int = 0): String = (
-    ( " " * indent )
-    ++ (if(needsUpdate) lib.red(show) else show)
-    ++ dependencies.map(
-      "\n" ++ _.dependencyTreeRecursion(indent + 1)
-    ).mkString
-  )
+  def dependencyTree: String = lib.dependencyTreeRecursion(this)
 }
 
 // TODO: all this hard codes the scala version, needs more flexibility
-class ScalaCompilerDependency(version: String)(implicit logger: Logger) extends BoundMavenDependency(MavenDependency("org.scala-lang","scala-compiler",version, Classifier.none), Seq(MavenRepository.central.url))
-class ScalaLibraryDependency (version: String)(implicit logger: Logger) extends BoundMavenDependency(MavenDependency("org.scala-lang","scala-library",version, Classifier.none), Seq(MavenRepository.central.url))
-class ScalaReflectDependency (version: String)(implicit logger: Logger) extends BoundMavenDependency(MavenDependency("org.scala-lang","scala-reflect",version, Classifier.none), Seq(MavenRepository.central.url))
+class ScalaCompilerDependency(cbtHasChanged: Boolean, mavenCache: File, version: String)(implicit logger: Logger) extends BoundMavenDependency(cbtHasChanged, mavenCache, MavenDependency("org.scala-lang","scala-compiler",version, Classifier.none), Seq(MavenResolver.central))
+class ScalaLibraryDependency (cbtHasChanged: Boolean, mavenCache: File, version: String)(implicit logger: Logger) extends BoundMavenDependency(cbtHasChanged, mavenCache, MavenDependency("org.scala-lang","scala-library",version, Classifier.none), Seq(MavenResolver.central))
+class ScalaReflectDependency (cbtHasChanged: Boolean, mavenCache: File, version: String)(implicit logger: Logger) extends BoundMavenDependency(cbtHasChanged, mavenCache, MavenDependency("org.scala-lang","scala-reflect",version, Classifier.none), Seq(MavenResolver.central))
 
-case class ScalaDependencies(version: String)(implicit val logger: Logger) extends Dependency{ sd =>
+case class ScalaDependencies(cbtHasChanged: Boolean, mavenCache: File, version: String)(implicit val logger: Logger) extends DependencyImplementation{ sd =>
   override final val needsUpdate = false
-  override def canBeCached = true
   def targetClasspath = ClassPath(Seq())
   def exportedClasspath = ClassPath(Seq())
-  def exportedJars = Seq[File]()  
   def dependencies = Seq(
-    new ScalaCompilerDependency(version),
-    new ScalaLibraryDependency(version),
-    new ScalaReflectDependency(version)
+    new ScalaCompilerDependency(cbtHasChanged, mavenCache, version),
+    new ScalaLibraryDependency(cbtHasChanged, mavenCache, version),
+    new ScalaReflectDependency(cbtHasChanged, mavenCache, version)
   )
 }
 
-case class BinaryDependency( path: File, dependencies: Seq[Dependency], canBeCached: Boolean )(implicit val logger: Logger) extends Dependency{
+case class BinaryDependency( path: File, dependencies: Seq[Dependency] )(implicit val logger: Logger) extends DependencyImplementation{
   def exportedClasspath = ClassPath(Seq(path))
-  def exportedJars = Seq[File](path)
   override def needsUpdate = false
   def targetClasspath = exportedClasspath
 }
 
 /** Allows to easily assemble a bunch of dependencies */
-case class Dependencies( dependencies: Seq[Dependency] )(implicit val logger: Logger) extends Dependency{
+case class Dependencies( dependencies: Seq[Dependency] )(implicit val logger: Logger) extends DependencyImplementation{
   override def needsUpdate = dependencies.exists(_.needsUpdate)
-  override def canBeCached = dependencies.forall(_.canBeCached)
   override def exportedClasspath = ClassPath(Seq())
-  override def exportedJars = Seq()
   override def targetClasspath = ClassPath(Seq())
 }
 object Dependencies{
   def apply( dependencies: Dependency* )(implicit logger: Logger): Dependencies = Dependencies( dependencies.to )
 }
 
-case class Stage1Dependency()(implicit val logger: Logger) extends Dependency{
-  override def needsUpdate = false // FIXME: think this through, might allow simplifications and/or optimizations
-  override def canBeCached = false
+case class Stage1Dependency(cbtHasChanged: Boolean, mavenCache: File, nailgunTarget: File, stage1Target: File, compatibilityTarget: File)(implicit val logger: Logger) extends DependencyImplementation{
+  override def needsUpdate = cbtHasChanged
   override def targetClasspath = exportedClasspath
   override def exportedClasspath = ClassPath( Seq(nailgunTarget, stage1Target) )
-  override def exportedJars = ???//Seq[File]()  
   override def dependencies = Seq(
-    MavenRepository.central.resolve(
+    CompatibilityDependency(cbtHasChanged, compatibilityTarget),
+    MavenResolver(cbtHasChanged,mavenCache,MavenResolver.central).resolve(
       MavenDependency("org.scala-lang","scala-library",constants.scalaVersion),
-      MavenDependency("org.scala-lang.modules","scala-xml_"+constants.scalaMajorVersion,"1.0.5")
+      MavenDependency("org.scala-lang.modules","scala-xml_"+constants.scalaMajorVersion,constants.scalaXmlVersion)
     )
   )
-  // FIXME: implement sanity check to prevent using incompatible scala-library and xml version on cp
-  override def classLoaderRecursion( latest: Map[(String,String),Dependency], cache: ClassLoaderCache ) = {
-    val a = actual( this, latest )
-    cache.transient.get(
-      a.classpath.string,
-      getClass.getClassLoader
-    )
-  }
 }
-case class CbtDependency()(implicit val logger: Logger) extends Dependency{
-  override def needsUpdate = false // FIXME: think this through, might allow simplifications and/or optimizations
-  override def canBeCached = false
+case class CompatibilityDependency(cbtHasChanged: Boolean, compatibilityTarget: File)(implicit val logger: Logger) extends DependencyImplementation{
+  override def needsUpdate = cbtHasChanged
+  override def targetClasspath = exportedClasspath
+  override def exportedClasspath = ClassPath( Seq(compatibilityTarget) )
+  override def dependencies = Seq()
+}
+case class CbtDependency(cbtHasChanged: Boolean, mavenCache: File, nailgunTarget: File, stage1Target: File, stage2Target: File, compatibilityTarget: File)(implicit val logger: Logger) extends DependencyImplementation{
+  override def needsUpdate = cbtHasChanged
   override def targetClasspath = exportedClasspath
   override def exportedClasspath = ClassPath( Seq( stage2Target ) )
-  override def exportedJars = ???
   override def dependencies = Seq(
-    Stage1Dependency(),
-    MavenRepository.central.resolve(
+    Stage1Dependency(cbtHasChanged, mavenCache, nailgunTarget, stage1Target, compatibilityTarget),
+    MavenResolver(cbtHasChanged, mavenCache,MavenResolver.central).resolve(
       MavenDependency("net.incongru.watchservice","barbary-watchservice","1.0"),
       MavenDependency("org.eclipse.jgit", "org.eclipse.jgit", "4.2.0.201601211800-r")
     )
@@ -254,16 +181,24 @@ abstract class DependenciesProxy{
 
 }
 class BoundMavenDependencies(
-  urls: Seq[URL], mavenDependencies: Seq[MavenDependency]
+  cbtHasChanged: Boolean, mavenCache: File, urls: Seq[URL], mavenDependencies: Seq[MavenDependency]
 )(implicit logger: Logger) extends Dependencies(
-  mavenDependencies.map( BoundMavenDependency(_,urls) )
+  mavenDependencies.map( BoundMavenDependency(cbtHasChanged,mavenCache,_,urls) )
 )
 case class MavenDependency(
   groupId: String, artifactId: String, version: String, classifier: Classifier = Classifier.none
-)
+){
+  private[cbt] def serialize = groupId ++ ":" ++ artifactId ++ ":"++ version ++ ":" ++ classifier.name.getOrElse("")
+}
+object MavenDependency{
+  private[cbt] def deserialize = (_:String).split(":") match {
+    case col => MavenDependency( col(0), col(1), col(2), Classifier(col.lift(3)) )
+  }
+}
+// FIXME: take MavenResolver instead of mavenCache and repositories separately
 case class BoundMavenDependency(
-  mavenDependency: MavenDependency, repositories: Seq[URL]
-)(implicit val logger: Logger) extends ArtifactInfo{
+  cbtHasChanged: Boolean, mavenCache: File, mavenDependency: MavenDependency, repositories: Seq[URL]
+)(implicit val logger: Logger) extends DependencyImplementation with ArtifactInfo{
   val MavenDependency( groupId, artifactId, version, classifier ) = mavenDependency
   assert(
     Option(groupId).collect{
@@ -283,14 +218,13 @@ case class BoundMavenDependency(
   )
 
   override def needsUpdate = false
-  override def canBeCached = dependencies.forall(_.canBeCached)
 
   private val groupPath = groupId.split("\\.").mkString("/")
   protected[cbt] def basePath = s"/$groupPath/$artifactId/$version/$artifactId-$version" ++ classifier.name.map("-"++_).getOrElse("")
   
   //private def coursierJarFile = userHome++"/.coursier/cache/v1/https/repo1.maven.org/maven2"++basePath++".jar"
 
-  override def exportedJars = Seq( jar )
+  def exportedJars = Seq( jar )
   override def exportedClasspath = ClassPath( exportedJars )
   override def targetClasspath = exportedClasspath
   import scala.collection.JavaConversions._
@@ -333,6 +267,8 @@ case class BoundMavenDependency(
     (pomXml \ "parent").collect{
       case parent =>
         BoundMavenDependency(
+          cbtHasChanged: Boolean,
+          mavenCache,
           MavenDependency(
             (parent \ "groupId").text,
             (parent \ "artifactId").text,
@@ -367,33 +303,36 @@ case class BoundMavenDependency(
 
   def dependencies: Seq[BoundMavenDependency] = {
     if(classifier == Classifier.sources) Seq()
-    else (pomXml \ "dependencies" \ "dependency").collect{
-      case xml if (xml \ "scope").text == "" && (xml \ "optional").text != "true" =>
-        val artifactId = lookup(xml,_ \ "artifactId").get
-        val groupId =
-          lookup(xml,_ \ "groupId").getOrElse(
-            dependencyVersions
-              .get(artifactId).map(_._1)
-              .getOrElse(
-                throw new Exception(s"$artifactId not found in \n$dependencyVersions")
+    else {
+      lib.cacheOnDisk(
+        cbtHasChanged, mavenCache ++ basePath ++ ".pom.dependencies"
+      )( MavenDependency.deserialize )( _.serialize ){
+        (pomXml \ "dependencies" \ "dependency").collect{
+          case xml if ( (xml \ "scope").text == "" || (xml \ "scope").text == "compile" ) && (xml \ "optional").text != "true" =>
+            val artifactId = lookup(xml,_ \ "artifactId").get
+            val groupId =
+              lookup(xml,_ \ "groupId").getOrElse(
+                dependencyVersions
+                  .get(artifactId).map(_._1)
+                  .getOrElse(
+                    throw new Exception(s"$artifactId not found in \n$dependencyVersions")
+                  )
               )
-          )
-        val version =
-          lookup(xml,_ \ "version").getOrElse(
-            dependencyVersions
-              .get(artifactId).map(_._2)
-              .getOrElse(
-                throw new Exception(s"$artifactId not found in \n$dependencyVersions")
+            val version =
+              lookup(xml,_ \ "version").getOrElse(
+                dependencyVersions
+                  .get(artifactId).map(_._2)
+                  .getOrElse(
+                    throw new Exception(s"$artifactId not found in \n$dependencyVersions")
+                  )
               )
-          )
-        BoundMavenDependency(
-          MavenDependency(
-            groupId, artifactId, version,
-            Classifier( Some( (xml \ "classifier").text ).filterNot(_ == "").filterNot(_ == null) )
-          ),
-          repositories
-        )
-    }.toVector
+            val classifier = Classifier( Some( (xml \ "classifier").text ).filterNot(_ == "").filterNot(_ == null) )
+            MavenDependency( groupId, artifactId, version, classifier )
+        }.toVector
+      }.map(
+        BoundMavenDependency( cbtHasChanged, mavenCache, _, repositories )
+      ).to
+    }
   }
   def lookup( xml: Node, accessor: Node => NodeSeq ): Option[String] = {
     //println("lookup in "++pomUrl)