diff options
| author | Andrew Or <andrewor14@gmail.com> | 2014-05-08 20:45:29 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-05-08 20:45:29 -0700 |
| commit | 8b7841299439b7dc590b2f7e2339f24e8f3e19f6 (patch) | |
| tree | 21db11b07ec32247e0efcaaa94b88e42d47b20f1 /pom.xml | |
| parent | 2fd2752e572921a9010614eb1c1238c493d34a7c (diff) | |
| download | spark-8b7841299439b7dc590b2f7e2339f24e8f3e19f6.tar.gz spark-8b7841299439b7dc590b2f7e2339f24e8f3e19f6.tar.bz2 spark-8b7841299439b7dc590b2f7e2339f24e8f3e19f6.zip | |
[SPARK-1755] Respect SparkSubmit --name on YARN
Right now, SparkSubmit ignores the `--name` flag for both yarn-client and yarn-cluster. This is a bug.
In client mode, SparkSubmit treats `--name` as a [cluster config](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L170) and does not propagate this to SparkContext.
In cluster mode, SparkSubmit passes this flag to `org.apache.spark.deploy.yarn.Client`, which only uses it for the [YARN ResourceManager](https://github.com/apache/spark/blob/master/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L80), but does not propagate this to SparkContext.
This PR ensures that `spark.app.name` is always set if SparkSubmit receives the `--name` flag, which is what the usage promises. This makes it possible for applications to start a SparkContext with an empty conf `val sc = new SparkContext(new SparkConf)`, and inherit the app name from SparkSubmit.
Tested both modes on a YARN cluster.
Author: Andrew Or <andrewor14@gmail.com>
Closes #699 from andrewor14/yarn-app-name and squashes the following commits:
98f6a79 [Andrew Or] Fix tests
dea932f [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-app-name
c86d9ca [Andrew Or] Respect SparkSubmit --name on YARN
Diffstat (limited to 'pom.xml')
0 files changed, 0 insertions, 0 deletions