diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2016-03-11 07:54:57 -0600 |
|---|---|---|
| committer | Tom Graves <tgraves@yahoo-inc.com> | 2016-03-11 07:54:57 -0600 |
| commit | 07f1c5447753a3d593cd6ececfcb03c11b1cf8ff (patch) | |
| tree | 74c4c9f81e64cc1ddde0b1c5e554a836808609e1 /core | |
| parent | 8fff0f92a4aca90b62c6e272eabcbb0257ba38d5 (diff) | |
| download | spark-07f1c5447753a3d593cd6ececfcb03c11b1cf8ff.tar.gz spark-07f1c5447753a3d593cd6ececfcb03c11b1cf8ff.tar.bz2 spark-07f1c5447753a3d593cd6ececfcb03c11b1cf8ff.zip | |
[SPARK-13577][YARN] Allow Spark jar to be multiple jars, archive.
In preparation for the demise of assemblies, this change allows the
YARN backend to use multiple jars and globs as the "Spark jar". The
config option has been renamed to "spark.yarn.jars" to reflect that.
A second option "spark.yarn.archive" was also added; if set, this
takes precedence and uploads an archive expected to contain the jar
files with the Spark code and its dependencies.
Existing deployments should keep working, mostly. This change drops
support for the "SPARK_JAR" environment variable, and also does not
fall back to using "jarOfClass" if no configuration is set, falling
back to finding files under SPARK_HOME instead. This should be fine
since "jarOfClass" probably wouldn't work unless you were using
spark-submit anyway.
Tested with the unit tests, and trying the different config options
on a YARN cluster.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #11500 from vanzin/SPARK-13577.
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/SparkConf.scala | 4 | ||||
| -rw-r--r-- | core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 1 |
2 files changed, 3 insertions, 2 deletions
diff --git a/core/src/main/scala/org/apache/spark/SparkConf.scala b/core/src/main/scala/org/apache/spark/SparkConf.scala index aaccf49eeb..ff8c631585 100644 --- a/core/src/main/scala/org/apache/spark/SparkConf.scala +++ b/core/src/main/scala/org/apache/spark/SparkConf.scala @@ -656,7 +656,9 @@ private[spark] object SparkConf extends Logging { "spark.memory.offHeap.enabled" -> Seq( AlternateConfig("spark.unsafe.offHeap", "1.6")), "spark.rpc.message.maxSize" -> Seq( - AlternateConfig("spark.akka.frameSize", "1.6")) + AlternateConfig("spark.akka.frameSize", "1.6")), + "spark.yarn.jars" -> Seq( + AlternateConfig("spark.yarn.jar", "2.0")) ) /** diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala index e8d0c3f9c3..4049fc0c41 100644 --- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala +++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala @@ -626,7 +626,6 @@ object SparkSubmit { val pathConfigs = Seq( "spark.jars", "spark.files", - "spark.yarn.jar", "spark.yarn.dist.files", "spark.yarn.dist.archives") pathConfigs.foreach { config => |