diff options
| author | Kay Ousterhout <kayousterhout@gmail.com> | 2014-06-26 08:20:27 -0500 |
|---|---|---|
| committer | Thomas Graves <tgraves@apache.org> | 2014-06-26 08:20:27 -0500 |
| commit | 48a82a827c99526b165c78d7e88faec43568a37a (patch) | |
| tree | da28a8c03507cc12a242d41201b35cb8ec9c8b25 /yarn | |
| parent | e4899a253728bfa7c78709a37a4837f74b72bd61 (diff) | |
| download | spark-48a82a827c99526b165c78d7e88faec43568a37a.tar.gz spark-48a82a827c99526b165c78d7e88faec43568a37a.tar.bz2 spark-48a82a827c99526b165c78d7e88faec43568a37a.zip | |
Remove use of spark.worker.instances
spark.worker.instances was added as part of this commit: https://github.com/apache/spark/commit/1617816090e7b20124a512a43860a21232ebf511
My understanding is that SPARK_WORKER_INSTANCES is supported for backwards compatibility,
but spark.worker.instances is never used (SparkSubmit.scala sets spark.executor.instances) so should
not have been added.
@sryza @pwendell @tgravescs LMK if I'm understanding this correctly
Author: Kay Ousterhout <kayousterhout@gmail.com>
Closes #1214 from kayousterhout/yarn_config and squashes the following commits:
3d7c491 [Kay Ousterhout] Remove use of spark.worker.instances
Diffstat (limited to 'yarn')
| -rw-r--r-- | yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala b/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala index 412dfe38d5..fd2694fe72 100644 --- a/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala +++ b/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala @@ -63,7 +63,7 @@ private[spark] class YarnClientSchedulerBackend( // variables. List(("--driver-memory", "SPARK_MASTER_MEMORY", "spark.master.memory"), ("--driver-memory", "SPARK_DRIVER_MEMORY", "spark.driver.memory"), - ("--num-executors", "SPARK_WORKER_INSTANCES", "spark.worker.instances"), + ("--num-executors", "SPARK_WORKER_INSTANCES", "spark.executor.instances"), ("--num-executors", "SPARK_EXECUTOR_INSTANCES", "spark.executor.instances"), ("--executor-memory", "SPARK_WORKER_MEMORY", "spark.executor.memory"), ("--executor-memory", "SPARK_EXECUTOR_MEMORY", "spark.executor.memory"), |