diff options
author | Andrew Or <andrewor14@gmail.com> | 2014-05-08 20:45:29 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-05-08 20:45:37 -0700 |
commit | 666bebe63ef30be80dd1496e5f9164dd4cdb2016 (patch) | |
tree | 1d77d73f1b760923bad6b18e9ec6f833454b771f /README.md | |
parent | ab912271a2f7ab267102c524755c30a9c1785cd0 (diff) | |
download | spark-666bebe63ef30be80dd1496e5f9164dd4cdb2016.tar.gz spark-666bebe63ef30be80dd1496e5f9164dd4cdb2016.tar.bz2 spark-666bebe63ef30be80dd1496e5f9164dd4cdb2016.zip |
[SPARK-1755] Respect SparkSubmit --name on YARN
Right now, SparkSubmit ignores the `--name` flag for both yarn-client and yarn-cluster. This is a bug.
In client mode, SparkSubmit treats `--name` as a [cluster config](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L170) and does not propagate this to SparkContext.
In cluster mode, SparkSubmit passes this flag to `org.apache.spark.deploy.yarn.Client`, which only uses it for the [YARN ResourceManager](https://github.com/apache/spark/blob/master/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L80), but does not propagate this to SparkContext.
This PR ensures that `spark.app.name` is always set if SparkSubmit receives the `--name` flag, which is what the usage promises. This makes it possible for applications to start a SparkContext with an empty conf `val sc = new SparkContext(new SparkConf)`, and inherit the app name from SparkSubmit.
Tested both modes on a YARN cluster.
Author: Andrew Or <andrewor14@gmail.com>
Closes #699 from andrewor14/yarn-app-name and squashes the following commits:
98f6a79 [Andrew Or] Fix tests
dea932f [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-app-name
c86d9ca [Andrew Or] Respect SparkSubmit --name on YARN
(cherry picked from commit 8b7841299439b7dc590b2f7e2339f24e8f3e19f6)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Diffstat (limited to 'README.md')
0 files changed, 0 insertions, 0 deletions