diff options
| author | jerryshao <saisai.shao@intel.com> | 2015-04-17 19:17:06 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-04-17 19:17:06 -0700 |
| commit | d850b4bd3a294dd245881e03f7f94bf970a7ee79 (patch) | |
| tree | 4babf14bc07ebe790c0b3d0b56a9cfb9708147f0 /yarn | |
| parent | 1991337336596f94698e79c2366f065c374128ab (diff) | |
| download | spark-d850b4bd3a294dd245881e03f7f94bf970a7ee79.tar.gz spark-d850b4bd3a294dd245881e03f7f94bf970a7ee79.tar.bz2 spark-d850b4bd3a294dd245881e03f7f94bf970a7ee79.zip | |
[SPARK-6975][Yarn] Fix argument validation error
`numExecutors` checking is failed when dynamic allocation is enabled with default configuration. Details can be seen is [SPARK-6975](https://issues.apache.org/jira/browse/SPARK-6975). sryza, please help me to review this, not sure is this the correct way, I think previous you change this part :)
Author: jerryshao <saisai.shao@intel.com>
Closes #5551 from jerryshao/SPARK-6975 and squashes the following commits:
4335da1 [jerryshao] Change according to the comments
77bdcbd [jerryshao] Fix argument validation error
Diffstat (limited to 'yarn')
| -rw-r--r-- | yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala | 8 |
1 files changed, 6 insertions, 2 deletions
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala index da6798cb1b..1423533470 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala @@ -103,9 +103,13 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf) * This is intended to be called only after the provided arguments have been parsed. */ private def validateArgs(): Unit = { - if (numExecutors <= 0) { + if (numExecutors < 0 || (!isDynamicAllocationEnabled && numExecutors == 0)) { throw new IllegalArgumentException( - "You must specify at least 1 executor!\n" + getUsageMessage()) + s""" + |Number of executors was $numExecutors, but must be at least 1 + |(or 0 if dynamic executor allocation is enabled). + |${getUsageMessage()} + """.stripMargin) } if (executorCores < sparkConf.getInt("spark.task.cpus", 1)) { throw new SparkException("Executor cores must not be less than " + |