[SPARK-8687] [YARN] Fix bug: Executor can't fetch the new set configuration in yarn-client

Spark initi the properties CoarseGrainedSchedulerBackend.start
```scala
    // TODO (prashant) send conf instead of properties
    driverEndpoint = rpcEnv.setupEndpoint(
      CoarseGrainedSchedulerBackend.ENDPOINT_NAME, new DriverEndpoint(rpcEnv, properties))
```
Then the yarn logic will set some configuration but not update in this `properties`.
So `Executor` won't gain the `properties`.

[Jira](https://issues.apache.org/jira/browse/SPARK-8687)

Author: huangzhaowei <carlmartinmax@gmail.com>

Closes #7066 from SaintBacchus/SPARK-8687 and squashes the following commits:

1de4f48 [huangzhaowei] Ensure all necessary properties have already been set before startup ExecutorLaucher

(cherry picked from commit 1b0c8e61040bf06213f9758f775679dcc41b0cce)
Signed-off-by: Andrew Or <andrew@databricks.com>

1 files changed, 6 insertions, 1 deletions

diff --git a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
index e7b0af6323..cb6008ead4 100644
--- a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
+++ b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
@@ -41,7 +41,6 @@ private[spark] class YarnClientSchedulerBackend(
    * This waits until the application is running.
    */
   override def start() {
-    super.start()
     val driverHost = conf.get("spark.driver.host")
     val driverPort = conf.get("spark.driver.port")
     val hostport = driverHost + ":" + driverPort
@@ -56,6 +55,12 @@ private[spark] class YarnClientSchedulerBackend(
     totalExpectedExecutors = args.numExecutors
     client = new Client(args, conf)
     appId = client.submitApplication()
+
+    // SPARK-8687: Ensure all necessary properties have already been set before
+    // we initialize our driver scheduler backend, which serves these properties
+    // to the executors
+    super.start()
+
     waitForApplication()
     monitorThread = asyncMonitorApplication()
     monitorThread.start()