diff options
| author | Xiangrui Meng <meng@databricks.com> | 2014-05-22 01:52:50 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2014-05-22 01:52:50 -0700 |
| commit | dba314029b4c9d72d7e48a2093b39edd01931f57 (patch) | |
| tree | 0f0efab82d551d1ed8901718a05c3b9d72db2c3b /assembly/src | |
| parent | 2a948e7e1a345ae4e3d89ea24123410819d799d1 (diff) | |
| download | spark-dba314029b4c9d72d7e48a2093b39edd01931f57.tar.gz spark-dba314029b4c9d72d7e48a2093b39edd01931f57.tar.bz2 spark-dba314029b4c9d72d7e48a2093b39edd01931f57.zip | |
[SPARK-1870] Make spark-submit --jars work in yarn-cluster mode.
Sent secondary jars to distributed cache of all containers and add the cached jars to classpath before executors start. Tested on a YARN cluster (CDH-5.0).
`spark-submit --jars` also works in standalone server and `yarn-client`. Thanks for @andrewor14 for testing!
I removed "Doesn't work for drivers in standalone mode with "cluster" deploy mode." from `spark-submit`'s help message, though we haven't tested mesos yet.
CC: @dbtsai @sryza
Author: Xiangrui Meng <meng@databricks.com>
Closes #848 from mengxr/yarn-classpath and squashes the following commits:
23e7df4 [Xiangrui Meng] rename spark.jar to __spark__.jar and app.jar to __app__.jar to avoid confliction apped $CWD/ and $CWD/* to the classpath remove unused methods
a40f6ed [Xiangrui Meng] standalone -> cluster
65e04ad [Xiangrui Meng] update spark-submit help message and add a comment for yarn-client
11e5354 [Xiangrui Meng] minor changes
3e7e1c4 [Xiangrui Meng] use sparkConf instead of hadoop conf
dc3c825 [Xiangrui Meng] add secondary jars to classpath in yarn
Diffstat (limited to 'assembly/src')
0 files changed, 0 insertions, 0 deletions