diff options
| author | Andrew <weiner.andrew.j@gmail.com> | 2016-01-27 09:31:44 +0000 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-01-27 09:31:44 +0000 |
| commit | 093291cf9b8729c0bd057cf67aed840b11f8c94a (patch) | |
| tree | 9956a0f9840b6eaae54cf668f0e2d680e202ec52 /docs/configuration.md | |
| parent | 90b0e562406a8bac529e190472e7f5da4030bf5c (diff) | |
| download | spark-093291cf9b8729c0bd057cf67aed840b11f8c94a.tar.gz spark-093291cf9b8729c0bd057cf67aed840b11f8c94a.tar.bz2 spark-093291cf9b8729c0bd057cf67aed840b11f8c94a.zip | |
[SPARK-1680][DOCS] Explain environment variables for running on YARN in cluster mode
JIRA 1680 added a property called spark.yarn.appMasterEnv. This PR draws users' attention to this special case by adding an explanation in configuration.html#environment-variables
Author: Andrew <weiner.andrew.j@gmail.com>
Closes #10869 from weineran/branch-yarn-docs.
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index d2a2f10524..74a8fb5d35 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1643,6 +1643,8 @@ to use on each machine and maximum memory. Since `spark-env.sh` is a shell script, some of these can be set programmatically -- for example, you might compute `SPARK_LOCAL_IP` by looking up the IP of a specific network interface. +Note: When running Spark on YARN in `cluster` mode, environment variables need to be set using the `spark.yarn.appMasterEnv.[EnvironmentVariableName]` property in your `conf/spark-defaults.conf` file. Environment variables that are set in `spark-env.sh` will not be reflected in the YARN Application Master process in `cluster` mode. See the [YARN-related Spark Properties](running-on-yarn.html#spark-properties) for more information. + # Configuring Logging Spark uses [log4j](http://logging.apache.org/log4j/) for logging. You can configure it by adding a |