Add sbt incremental compilation support

To test this with sbt, see
https://github.com/lampepfl/dotty/wiki/Using-Dotty-with-sbt

The following flags are added:
 - -Yforce-sbt-phases: Run the phases used by sbt for incremental compilation
   (ExtractDependencies and ExtractAPI) even if the compiler is ran outside of
   sbt, for debugging.
 - -Ydump-sbt-inc: For every compiled foo.scala, output the API
   representation and dependencies used for sbt incremental compilation
   in foo.inc, implies -Yforce-sbt-phases.

This commit introduces two new phases which do not transform trees:
- `ExtractDependencies` which extracts the dependency information of the current
  compilation unit and sends it to sbt via callbacks
- `ExtractAPI` which creates a representation of the API of the current compilation
  unit and sends it to sbt via callbacks

Briefly, when a file changes sbt will recompile it, if its API has
changed (determined by what `ExtractAPI` sent) then sbt will determine
which reverse-dependencies (determined by what `ExtractDependencies`
sent) of the API have to be recompiled depending on what changed.
See http://www.scala-sbt.org/0.13/docs/Understanding-Recompilation.html for
more information on how sbt incremental compilation works.

This phase was originally based on
https://github.com/adriaanm/scala/tree/sbt-api-consolidate/src/compiler/scala/tools/sbt
which attempts to integrate the sbt phases into scalac (and is itself based
on https://github.com/sbt/sbt/tree/0.13/compile/interface/src/main/scala/xsbt),
but it has been heavily refactored and adapted to Dotty. The main
functional differences are:
- ExtractDependencies runs right after Frontend (so that we don't lose
  dependency informations because of the simplifications done by PostTyper),
  but ExtractAPI runs right after PostTyper (so that SuperAccessors are
  part of the API).
- `ExtractAPI` only extract types as they are defined and never "as seen
  from" some some specific prefix, see its documentation for more details.
- `ExtractDependenciesTraverser` and `ExtractUsedNames` have been fused into
  one tree traversal in `ExtractDependenciesCollector`.

TODO: Try to run these phases in parallel with the rest of the compiler
pipeline since they're independent (except for the sbt callbacks in `GenBCode`) ?

1 files changed, 61 insertions, 0 deletions

diff --git a/src/dotty/tools/dotc/sbt/ThunkHolder.scala b/src/dotty/tools/dotc/sbt/ThunkHolder.scala
new file mode 100644
index 000000000..e377de6da
--- /dev/null
+++ b/src/dotty/tools/dotc/sbt/ThunkHolder.scala
@@ -0,0 +1,61 @@
+package dotty.tools.dotc
+package sbt
+
+import scala.annotation.tailrec
+import scala.collection.mutable.ListBuffer
+import xsbti.api
+
+/** Create and hold thunks. A thunk is a (potentially) unevaluated value
+ *  that may be evaluated once.
+ */
+private[sbt] trait ThunkHolder {
+  private[this] val thunks = new ListBuffer[api.Lazy[_]]
+
+  /** Force all unevaluated thunks to prevent space leaks. */
+  @tailrec protected final def forceThunks(): Unit = if (!thunks.isEmpty) {
+    val toForce = thunks.toList
+    thunks.clear()
+    toForce.foreach(_.get())
+    // Forcing thunks may create new thunks
+    forceThunks()
+  }
+
+  /** Store the by-name parameter `s` in a `Lazy` container without evaluating it.
+   *  It will be forced by the next call to `forceThunks()`
+   */
+  def lzy[T <: AnyRef](t: => T): api.Lazy[T] = {
+    val l = SafeLazy(() => t)
+    thunks += l
+    l
+  }
+
+  /** Store the parameter `s` in a `Lazy` container, since `s` is not by-name, there
+   *  is nothing to force.
+   *
+   *  TODO: Get rid of this method. It is only needed because some xsbti.api classes
+   *  take lazy arguments when they could be strict, but this can be fixed in sbt,
+   *  see https://github.com/sbt/zinc/issues/114
+   */
+  def strict2lzy[T <: AnyRef](t: T): api.Lazy[T] =
+    SafeLazy.strict(t)
+}
+
+// TODO: Use xsbti.SafeLazy once https://github.com/sbt/zinc/issues/113 is fixed
+private object SafeLazy {
+  def apply[T <: AnyRef](eval: () => T): xsbti.api.Lazy[T] =
+    new Impl(eval)
+
+  def strict[T <: AnyRef](value: T): xsbti.api.Lazy[T] =
+    new Strict(value)
+
+  private[this] final class Impl[T <: AnyRef](private[this] var eval: () => T) extends xsbti.api.AbstractLazy[T] {
+    private[this] lazy val _t = {
+      val t = eval()
+      eval = null // clear the reference, ensuring the only memory we hold onto is the result
+      t
+    }
+    def get: T = _t
+  }
+
+  private[this] final class Strict[T <: AnyRef](val get: T) extends xsbti.api.Lazy[T] with java.io.Serializable
+}