diff options
author | huangzhaowei <carlmartinmax@gmail.com> | 2015-07-01 23:14:13 -0700 |
---|---|---|
committer | Andrew Or <andrew@databricks.com> | 2015-07-01 23:14:13 -0700 |
commit | 1b0c8e61040bf06213f9758f775679dcc41b0cce (patch) | |
tree | 65a891546301b44143b8ee189d3b6bd65630ffcd /yarn | |
parent | 3697232b7d438979cc119b2a364296b0eec4a16a (diff) | |
download | spark-1b0c8e61040bf06213f9758f775679dcc41b0cce.tar.gz spark-1b0c8e61040bf06213f9758f775679dcc41b0cce.tar.bz2 spark-1b0c8e61040bf06213f9758f775679dcc41b0cce.zip |
[SPARK-8687] [YARN] Fix bug: Executor can't fetch the new set configuration in yarn-client
Spark initi the properties CoarseGrainedSchedulerBackend.start
```scala
// TODO (prashant) send conf instead of properties
driverEndpoint = rpcEnv.setupEndpoint(
CoarseGrainedSchedulerBackend.ENDPOINT_NAME, new DriverEndpoint(rpcEnv, properties))
```
Then the yarn logic will set some configuration but not update in this `properties`.
So `Executor` won't gain the `properties`.
[Jira](https://issues.apache.org/jira/browse/SPARK-8687)
Author: huangzhaowei <carlmartinmax@gmail.com>
Closes #7066 from SaintBacchus/SPARK-8687 and squashes the following commits:
1de4f48 [huangzhaowei] Ensure all necessary properties have already been set before startup ExecutorLaucher
Diffstat (limited to 'yarn')
-rw-r--r-- | yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala | 7 |
1 files changed, 6 insertions, 1 deletions
diff --git a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala index dd8c4fdb54..3a0b9443d2 100644 --- a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala +++ b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala @@ -41,7 +41,6 @@ private[spark] class YarnClientSchedulerBackend( * This waits until the application is running. */ override def start() { - super.start() val driverHost = conf.get("spark.driver.host") val driverPort = conf.get("spark.driver.port") val hostport = driverHost + ":" + driverPort @@ -56,6 +55,12 @@ private[spark] class YarnClientSchedulerBackend( totalExpectedExecutors = args.numExecutors client = new Client(args, conf) appId = client.submitApplication() + + // SPARK-8687: Ensure all necessary properties have already been set before + // we initialize our driver scheduler backend, which serves these properties + // to the executors + super.start() + waitForApplication() monitorThread = asyncMonitorApplication() monitorThread.start() |