diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2015-05-01 21:20:46 +0100 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2015-05-01 21:20:46 +0100 |
| commit | 7b5dd3e3c0030087eea5a8224789352c03717c1d (patch) | |
| tree | 79c5fc60f9288d31d424a6a9e83ac34bb90ca6b8 | |
| parent | f53a48827ef024f91b292132075e5598c9cb94bb (diff) | |
| download | spark-7b5dd3e3c0030087eea5a8224789352c03717c1d.tar.gz spark-7b5dd3e3c0030087eea5a8224789352c03717c1d.tar.bz2 spark-7b5dd3e3c0030087eea5a8224789352c03717c1d.zip | |
[SPARK-7281] [YARN] Add option to set AM's lib path in client mode.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #5813 from vanzin/SPARK-7281 and squashes the following commits:
1cb6f42 [Marcelo Vanzin] [SPARK-7281] [yarn] Add option to set AM's lib path in client mode.
| -rw-r--r-- | docs/running-on-yarn.md | 7 | ||||
| -rw-r--r-- | yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 4 |
2 files changed, 11 insertions, 0 deletions
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 0968fc5ad6..b6701b64c2 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -190,6 +190,13 @@ Most of the configs are the same for Spark on YARN as for other deployment modes </td> </tr> <tr> + <td><code>spark.yarn.am.extraLibraryPath</code></td> + <td>(none)</td> + <td> + Set a special library path to use when launching the application master in client mode. + </td> +</tr> +<tr> <td><code>spark.yarn.maxAppAttempts</code></td> <td>yarn.resourcemanager.am.max-attempts in YARN</td> <td> diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala index b945395f24..31ab6b491e 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala @@ -544,6 +544,10 @@ private[spark] class Client( } javaOpts ++= Utils.splitCommandString(opts).map(YarnSparkHadoopUtil.escapeForShell) } + + sparkConf.getOption("spark.yarn.am.extraLibraryPath").foreach { paths => + prefixEnv = Some(Utils.libraryPathEnvPrefix(Seq(paths))) + } } // For log4j configuration to reference |