aboutsummaryrefslogtreecommitdiff
path: root/core
diff options
context:
space:
mode:
authorMatei Zaharia <matei@databricks.com>2014-05-19 18:42:28 -0700
committerTathagata Das <tathagata.das1565@gmail.com>2014-05-19 18:42:28 -0700
commit5af99d7617ba3b9fbfdb345ef9571b7dd41f45a1 (patch)
tree2d0e90945da59eaa5df41221027c365a823895bf /core
parent1811ba8ccb580979aa2e12019e6a82805f09ab53 (diff)
downloadspark-5af99d7617ba3b9fbfdb345ef9571b7dd41f45a1.tar.gz
spark-5af99d7617ba3b9fbfdb345ef9571b7dd41f45a1.tar.bz2
spark-5af99d7617ba3b9fbfdb345ef9571b7dd41f45a1.zip
SPARK-1879. Increase MaxPermSize since some of our builds have many classes
See https://issues.apache.org/jira/browse/SPARK-1879 -- builds with Hadoop2 and Hive ran out of PermGen space in spark-shell, when those things added up with the Scala compiler. Note that users can still override it by setting their own Java options with this change. Their options will come later in the command string than the -XX:MaxPermSize=128m. Author: Matei Zaharia <matei@databricks.com> Closes #823 from mateiz/spark-1879 and squashes the following commits: 6bc0ee8 [Matei Zaharia] Increase MaxPermSize to 128m since some of our builds have lots of classes
Diffstat (limited to 'core')
-rw-r--r--core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala4
1 files changed, 3 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala b/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
index c7f0f244ea..4af5bc3afa 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
@@ -65,6 +65,8 @@ object CommandUtils extends Logging {
Seq()
}
+ val permGenOpt = Seq("-XX:MaxPermSize=128m")
+
// Figure out our classpath with the external compute-classpath script
val ext = if (System.getProperty("os.name").startsWith("Windows")) ".cmd" else ".sh"
val classPath = Utils.executeAndGetOutput(
@@ -73,7 +75,7 @@ object CommandUtils extends Logging {
val userClassPath = command.classPathEntries ++ Seq(classPath)
Seq("-cp", userClassPath.filterNot(_.isEmpty).mkString(File.pathSeparator)) ++
- libraryOpts ++ extraOpts ++ workerLocalOpts ++ memoryOpts
+ permGenOpt ++ libraryOpts ++ extraOpts ++ workerLocalOpts ++ memoryOpts
}
/** Spawn a thread that will redirect a given stream to a file */