diff options
| author | Kanwaljit Singh <kanwaljit.singh@guavus.com> | 2014-12-19 19:25:39 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2014-12-19 19:27:23 -0800 |
| commit | 1d648123a77bbcd9b7a34cc0d66c14fa85edfecd (patch) | |
| tree | 7e45a9a669845fe3ef340fc0c3f20797a033117e /core | |
| parent | 8d932475e6759e869c16ce6cac203a2e56558716 (diff) | |
| download | spark-1d648123a77bbcd9b7a34cc0d66c14fa85edfecd.tar.gz spark-1d648123a77bbcd9b7a34cc0d66c14fa85edfecd.tar.bz2 spark-1d648123a77bbcd9b7a34cc0d66c14fa85edfecd.zip | |
SPARK-2641: Passing num executors to spark arguments from properties file
Since we can set spark executor memory and executor cores using property file, we must also be allowed to set the executor instances.
Author: Kanwaljit Singh <kanwaljit.singh@guavus.com>
Closes #1657 from kjsingh/branch-1.0 and squashes the following commits:
d8a5a12 [Kanwaljit Singh] SPARK-2641: Fixing how spark arguments are loaded from properties file for num executors
Conflicts:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala index f0e9ee67f6..958bd03617 100644 --- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala +++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala @@ -120,6 +120,8 @@ private[spark] class SparkSubmitArguments(args: Seq[String], env: Map[String, St name = Option(name).orElse(sparkProperties.get("spark.app.name")).orNull jars = Option(jars).orElse(sparkProperties.get("spark.jars")).orNull deployMode = Option(deployMode).orElse(env.get("DEPLOY_MODE")).orNull + numExecutors = Option(numExecutors) + .getOrElse(defaultProperties.get("spark.executor.instances").orNull) // Try to set main class from JAR if no --class argument is given if (mainClass == null && !isPython && primaryResource != null) { |