Integrate flat classpath with the compiler

This commit integrates with the compiler the whole flat classpath
representation build next to the recursive one as an alternative.

From now flat classpath really works and can be turned on. There's
added flag -YclasspathImpl with two options: recursive (the default
one) and flat.

It was needed to make the dynamic dispatch to the particular
classpath representation according to the chosen type of a classpath
representation.

There's added PathResolverFactory which is used instead of a concrete
implementation of a path resolver. It turned out that only a small
subset of path resolvers methods is used outside this class in Scala
sources. Therefore, PathResolverFactory returns an instance of a base
interface PathResolverResult providing only these used methods.

PathResolverFactory in combination with matches in some other places
ensures that in all places using classpath we create/get the proper
representation.

Also the classPath method in Global is modified to use the dynamic
dispatch. This is very important change as a return type changed to
the base ClassFileLookup providing subset of old ClassPath public
methods. It can be problematic if someone was using in his project
the explicit ClassPath type or public methods which are not provided
via ClassFileLookup. I tested flat classpath with sbt and Scala IDE
and there were no problems. Also was looking at sources of some other
projects like e.g. Scala plugin for IntelliJ and there shouldn't be
problems, I think, but it would be better to check these changes
using the community build.

Scalap's Main.scala is changed to be able to use both implementations
and also to use flags related to the classpath implementation.

The classpath invalidation is modified to work properly with the old
(recursive) classpath representation after changes made in a Global.
In the case of the attempt to use the invalidation for the flat cp it
just throws exception with a message that the flat one currently
doesn't support the invalidation. And also that's why the partest's
test for the invalidation has been changed to use (always) the old
implementation. There's added an adequate comment with TODO to this
file.

There's added partest test generating various dependencies
(directories, zips and jars with sources and class files) and testing
whether the compilation and further running an application works
correctly, when there are these various types of entries specified as
-classpath and -sourcepath. It should be a good approximation of real
use cases.

4 files changed, 273 insertions, 9 deletions

diff --git a/test/files/run/t6502.scala b/test/files/run/t6502.scala
index ced1b5812d..4ce034a482 100644
--- a/test/files/run/t6502.scala
+++ b/test/files/run/t6502.scala
@@ -1,6 +1,7 @@
-import scala.tools.partest._
-import java.io.File
+import scala.tools.nsc.Settings
 import scala.tools.nsc.interpreter.ILoop
+import scala.tools.nsc.settings.ClassPathRepresentationType
+import scala.tools.partest._
 
 object Test extends StoreReporterDirectTest {
   def code = ???
@@ -10,6 +11,14 @@ object Test extends StoreReporterDirectTest {
     compileString(newCompiler("-cp", classpath, "-d", s"${testOutput.path}/$jarFileName"))(code)
   }
 
+  // TODO flat classpath doesn't support the classpath invalidation yet so we force using the recursive one
+  // it's the only test which needed such a workaround
+  override def settings = {
+    val settings = new Settings
+    settings.YclasspathImpl.value = ClassPathRepresentationType.Recursive
+    settings
+  }
+
   def app1 = """
     package test
 
@@ -41,7 +50,8 @@ object Test extends StoreReporterDirectTest {
     val jar = "test1.jar"
     compileCode(app1, jar)
 
-    val output = ILoop.run(List(s":require ${testOutput.path}/$jar", "test.Test.test()"))
+    val codeToRun = toCodeInSeparateLines(s":require ${testOutput.path}/$jar", "test.Test.test()")
+    val output = ILoop.run(codeToRun, settings)
     val lines  = output.split("\n")
     val res1   = lines(4).contains("Added") && lines(4).contains("test1.jar")
     val res2   = lines(lines.length-3).contains("testing...")
@@ -56,7 +66,8 @@ object Test extends StoreReporterDirectTest {
     val jar2 = "test2.jar"
     compileCode(app2, jar2)
 
-    val output = ILoop.run(List(s":require ${testOutput.path}/$jar1", s":require ${testOutput.path}/$jar2"))
+    val codeToRun = toCodeInSeparateLines(s":require ${testOutput.path}/$jar1", s":require ${testOutput.path}/$jar2")
+    val output = ILoop.run(codeToRun, settings)
     val lines  = output.split("\n")
     val res1   = lines(4).contains("Added") && lines(4).contains("test1.jar")
     val res2   = lines(lines.length-3).contains("test2.jar") && lines(lines.length-3).contains("existing classpath entries conflict")
@@ -71,7 +82,8 @@ object Test extends StoreReporterDirectTest {
     val jar3 = "test3.jar"
     compileCode(app3, jar3)
 
-    val output = ILoop.run(List(s":require ${testOutput.path}/$jar1", s":require ${testOutput.path}/$jar3", "test.Test3.test()"))
+    val codeToRun = toCodeInSeparateLines(s":require ${testOutput.path}/$jar1", s":require ${testOutput.path}/$jar3", "test.Test3.test()")
+    val output = ILoop.run(codeToRun, settings)
     val lines  = output.split("\n")
     val res1   = lines(4).contains("Added") && lines(4).contains("test1.jar")
     val res2   = lines(lines.length-3).contains("new object in existing package")
@@ -83,7 +95,8 @@ object Test extends StoreReporterDirectTest {
   def test4(): Unit = {
     // twice the same jar should be rejected
     val jar1   = "test1.jar"
-    val output = ILoop.run(List(s":require ${testOutput.path}/$jar1", s":require ${testOutput.path}/$jar1"))
+    val codeToRun = toCodeInSeparateLines(s":require ${testOutput.path}/$jar1", s":require ${testOutput.path}/$jar1")
+    val output = ILoop.run(codeToRun, settings)
     val lines  = output.split("\n")
     val res1   = lines(4).contains("Added") && lines(4).contains("test1.jar")
     val res2   = lines(lines.length-3).contains("test1.jar") && lines(lines.length-3).contains("existing classpath entries conflict")
@@ -98,4 +111,6 @@ object Test extends StoreReporterDirectTest {
     test3()
     test4()
   }
+
+  def toCodeInSeparateLines(lines: String*): String = lines.map(_ + "\n").mkString
 }
diff --git a/test/files/run/various-flat-classpath-types.check b/test/files/run/various-flat-classpath-types.check
new file mode 100644
index 0000000000..401f707d0e
--- /dev/null
+++ b/test/files/run/various-flat-classpath-types.check
@@ -0,0 +1,12 @@
+ZipBin()
+JarBin()
+DirBin()
+ZipSrc()
+JarSrc()
+DirSrc()
+NestedZipBin()
+NestedJarBin()
+NestedDirBin()
+NestedZipSrc()
+NestedJarSrc()
+NestedDirSrc()
\ No newline at end of file
diff --git a/test/files/run/various-flat-classpath-types.scala b/test/files/run/various-flat-classpath-types.scala
new file mode 100644
index 0000000000..d39019e885
--- /dev/null
+++ b/test/files/run/various-flat-classpath-types.scala
@@ -0,0 +1,214 @@
+/*
+ * Copyright (c) 2014 Contributor. All rights reserved.
+ */
+
+import java.io.{File => JFile, FileInputStream, FileOutputStream}
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.reflect.io.{Directory, File}
+import scala.tools.nsc.classpath.FlatClassPath.RootPackage
+import scala.tools.nsc.classpath.PackageNameUtils
+import scala.tools.nsc.io.Jar
+
+/**
+ * Generates directories, jars and zip files containing sources and classes
+ * (the result of a compilation which is executed here)
+ * and use them as a class- and sourcepath during compilation and running
+ * created application. At the end everything is cleaned up.
+ *
+ * It can test also current, recursive classpath. Just right now we force
+ * flat classpath to test it also when the recursive one would be set as a default.
+ */
+object Test {
+
+  private implicit class JFileOps(file: JFile) {
+
+    def createDir(newDirName: String) = {
+      val newDir = new JFile(file, newDirName)
+      newDir.mkdir()
+      newDir
+    }
+
+    def createSrcFile(newFileName: String) = createFile(newFileName + ".scala")
+
+    def createFile(fullFileName: String) = {
+      val newFile = new JFile(file, fullFileName)
+      newFile.createNewFile()
+      newFile
+    }
+
+    def writeAll(text: String): Unit = File(file) writeAll text
+
+    def moveContentToZip(zipName: String): Unit = {
+      val newZip = zipsDir createFile s"$zipName.zip"
+      val outputStream = new ZipOutputStream(new FileOutputStream(newZip))
+
+      def addFileToZip(dirPrefix: String = "")(fileToAdd: JFile): Unit =
+        if (fileToAdd.isDirectory) {
+          val dirEntryName = fileToAdd.getName + "/"
+          outputStream.putNextEntry(new ZipEntry(dirEntryName))
+          fileToAdd.listFiles() foreach addFileToZip(dirEntryName)
+        } else {
+          val inputStream = new FileInputStream(fileToAdd)
+          outputStream.putNextEntry(new ZipEntry(dirPrefix + fileToAdd.getName))
+
+          val buffer = new Array[Byte](1024)
+          var count = inputStream.read(buffer)
+          while (count > 0) {
+            outputStream.write(buffer, 0, count)
+            count = inputStream.read(buffer)
+          }
+
+          inputStream.close()
+        }
+
+      file.listFiles() foreach addFileToZip()
+      outputStream.close()
+
+      cleanDir(file)
+    }
+
+    def moveContentToJar(jarName: String): Unit = {
+      val newJar = jarsDir createFile s"$jarName.jar"
+      Jar.create(file = File(newJar), sourceDir = Directory(file), mainClass = "won't be used")
+      cleanDir(file)
+    }
+
+    def path: String = file.getAbsolutePath
+  }
+
+  private case class DirRep(name: String, nestedDirs: Seq[DirRep] = Nil, sourceFiles: Seq[String] = Nil)
+
+  private val compiler = new scala.tools.nsc.MainClass
+  private val appRunner = new scala.tools.nsc.MainGenericRunner
+  private val classPathImplFlag = "-YclasspathImpl:flat"
+  private val javaClassPath = sys.props("java.class.path")
+
+  // creates a test dir in a temporary dir containing compiled files of this test
+  // root dir will be automatically deleted after the end of test
+  private val rootDir = new JFile(sys.props("partest.output"))
+  private val testDir = rootDir createDir s"cp-tests-${System.currentTimeMillis()}"
+
+  private val jarsDir = testDir createDir "jars"
+  private val zipsDir = testDir createDir "zips"
+  private val srcDir = testDir createDir "src"
+  private val binDir = testDir createDir "bin"
+  private val outDir = testDir createDir "out"
+
+  def main(args: Array[String]): Unit = {
+    createClassesZipInZipsDir()
+    createClassesJarInJarsDir()
+    createClassesInBinDir()
+    createSourcesZipInZipsDir()
+    createSourcesJarInJarsDir()
+    createSourcesInSrcDir()
+    compileFinalApp()
+    runApp()
+    // at the end all created files will be deleted automatically
+  }
+
+  private def createClassesZipInZipsDir(): Unit = {
+    val baseFileName = "ZipBin"
+    createStandardSrcHierarchy(baseFileName)
+    compileSrc(baseFileName)
+    outDir moveContentToZip "Bin"
+    cleanDir(srcDir)
+  }
+
+  private def createClassesJarInJarsDir(): Unit = {
+    val baseFileName = "JarBin"
+    createStandardSrcHierarchy(baseFileName)
+    compileSrc(baseFileName)
+    outDir moveContentToJar "Bin"
+    cleanDir(srcDir)
+  }
+
+  private def createClassesInBinDir(): Unit = {
+    val baseFileName = "DirBin"
+    createStandardSrcHierarchy(baseFileName)
+    compileSrc(baseFileName, destination = binDir)
+    cleanDir(srcDir)
+  }
+
+  private def createSourcesZipInZipsDir(): Unit = {
+    createStandardSrcHierarchy(baseFileName = "ZipSrc")
+    srcDir moveContentToZip "Src"
+  }
+
+  private def createSourcesJarInJarsDir(): Unit = {
+    createStandardSrcHierarchy(baseFileName = "JarSrc")
+    srcDir moveContentToJar "Src"
+  }
+
+  private def createSourcesInSrcDir(): Unit = {
+    createStandardSrcHierarchy(baseFileName = "DirSrc")
+
+    val appFile = srcDir createSrcFile "Main"
+    appFile writeAll s"""import nested._
+         | object Main extends App {
+         |   println(new ZipBin)
+         |   println(new JarBin)
+         |   println(new DirBin)
+         |   println(new ZipSrc)
+         |   println(new JarSrc)
+         |   println(new DirSrc)
+         |
+         |   println(new NestedZipBin)
+         |   println(new NestedJarBin)
+         |   println(new NestedDirBin)
+         |   println(new NestedZipSrc)
+         |   println(new NestedJarSrc)
+         |   println(new NestedDirSrc)
+         | }
+       """.stripMargin
+  }
+
+  private def compileFinalApp(): Unit = {
+    val classPath = mkPath(javaClassPath, binDir.path, zipsDir.path + "/Bin.zip", jarsDir.path + "/Bin.jar")
+    val sourcePath = mkPath(srcDir.path, zipsDir.path + "/Src.zip", jarsDir.path + "/Src.jar")
+
+    compiler.process(Array(classPathImplFlag, "-cp", classPath, "-sourcepath", sourcePath,
+      "-d", outDir.path, s"${srcDir.path}/Main.scala"))
+  }
+
+  private def runApp(): Unit = {
+    val classPath = mkPath(javaClassPath, outDir.path, binDir.path, zipsDir.path + "/Bin.zip", jarsDir.path + "/Bin.jar")
+    appRunner.process(Array(classPathImplFlag, "-cp", classPath, "Main"))
+  }
+
+  private def createStandardSrcHierarchy(baseFileName: String): Unit =
+    createSources(RootPackage, srcDir,
+      DirRep("",
+        nestedDirs = Seq(DirRep("nested", sourceFiles = Seq("Nested" + baseFileName))),
+        sourceFiles = Seq(baseFileName)
+      )
+    )
+
+  private def createSources(pkg: String, dirFile: JFile, dirRep: DirRep): Unit = {
+    dirRep.nestedDirs foreach { rep =>
+      val nestedDir = dirFile createDir rep.name
+      val nestedPkg = PackageNameUtils.packagePrefix(pkg) + rep.name
+      createSources(nestedPkg, nestedDir, rep)
+    }
+
+    val pkgHeader = if (pkg == RootPackage) "" else s"package $pkg\n\n"
+    dirRep.sourceFiles foreach { srcName =>
+      val text = s"""${pkgHeader}case class $srcName(x: String = "")"""
+      val srcFile = dirFile createSrcFile srcName
+      srcFile writeAll text
+    }
+  }
+
+  private def compileSrc(baseFileName: String, destination: JFile = outDir): Unit = {
+    val srcDirPath = srcDir.path
+    compiler.process(Array(classPathImplFlag, "-cp", javaClassPath, "-d", destination.path,
+      s"$srcDirPath/$baseFileName.scala", s"$srcDirPath/nested/Nested$baseFileName.scala"))
+  }
+
+  private def cleanDir(dir: JFile): Unit =
+    dir.listFiles().foreach { file =>
+      if (file.isDirectory) cleanDir(file)
+      file.delete()
+    }
+
+  private def mkPath(pathEntries: String*) = pathEntries.mkString(File.pathSeparator)
+}
diff --git a/test/junit/scala/tools/nsc/symtab/SymbolTableForUnitTesting.scala b/test/junit/scala/tools/nsc/symtab/SymbolTableForUnitTesting.scala
index e4be42ac96..d61c62784c 100644
--- a/test/junit/scala/tools/nsc/symtab/SymbolTableForUnitTesting.scala
+++ b/test/junit/scala/tools/nsc/symtab/SymbolTableForUnitTesting.scala
@@ -3,6 +3,9 @@ package symtab
 
 import scala.reflect.ClassTag
 import scala.reflect.internal.{Phase, NoPhase, SomePhase}
+import scala.tools.nsc.classpath.FlatClassPath
+import scala.tools.nsc.settings.ClassPathRepresentationType
+import scala.tools.util.FlatClassPathResolver
 import scala.tools.util.PathResolver
 import util.ClassPath
 import io.AbstractFile
@@ -26,13 +29,28 @@ class SymbolTableForUnitTesting extends SymbolTable {
   class LazyTreeCopier extends super.LazyTreeCopier with TreeCopier
 
   override def isCompilerUniverse: Boolean = true
-  def classPath = new PathResolver(settings).result
+
+  def classPath = platform.classPath
+  def flatClassPath: FlatClassPath = platform.flatClassPath
 
   object platform extends backend.Platform {
     val symbolTable: SymbolTableForUnitTesting.this.type = SymbolTableForUnitTesting.this
     lazy val loaders: SymbolTableForUnitTesting.this.loaders.type = SymbolTableForUnitTesting.this.loaders
+
     def platformPhases: List[SubComponent] = Nil
-    val classPath: ClassPath[AbstractFile] = new PathResolver(settings).result
+
+    lazy val classPath: ClassPath[AbstractFile] = {
+      assert(settings.YclasspathImpl.value == ClassPathRepresentationType.Recursive,
+        "It's not possible to use the recursive classpath representation, when it's not the chosen classpath scanning method")
+      new PathResolver(settings).result
+    }
+
+    private[nsc] lazy val flatClassPath: FlatClassPath = {
+      assert(settings.YclasspathImpl.value == ClassPathRepresentationType.Flat,
+        "It's not possible to use the flat classpath representation, when it's not the chosen classpath scanning method")
+      new FlatClassPathResolver(settings).result
+    }
+
     def isMaybeBoxed(sym: Symbol): Boolean = ???
     def needCompile(bin: AbstractFile, src: AbstractFile): Boolean = ???
     def externalEquals: Symbol = ???
@@ -50,7 +68,12 @@ class SymbolTableForUnitTesting extends SymbolTable {
 
   class GlobalMirror extends Roots(NoSymbol) {
     val universe: SymbolTableForUnitTesting.this.type = SymbolTableForUnitTesting.this
-    def rootLoader: LazyType = new loaders.PackageLoader(classPath)
+
+    def rootLoader: LazyType = settings.YclasspathImpl.value match {
+      case ClassPathRepresentationType.Flat => new loaders.PackageLoaderUsingFlatClassPath(FlatClassPath.RootPackage, flatClassPath)
+      case ClassPathRepresentationType.Recursive => new loaders.PackageLoader(classPath)
+    }
+
     override def toString = "compiler mirror"
   }