diff options
| author | Sandy Ryza <sandy@cloudera.com> | 2014-02-26 10:00:02 -0600 |
|---|---|---|
| committer | Thomas Graves <tgraves@apache.org> | 2014-02-26 10:00:02 -0600 |
| commit | b8a1871953058c67b49b7f8455cbb417d5b50ab6 (patch) | |
| tree | 9c1029df66bb11cadbc6308550a6f1b559f92346 /docs/running-on-yarn.md | |
| parent | c852201ce95c7c982ff3794c114427eb33e92922 (diff) | |
| download | spark-b8a1871953058c67b49b7f8455cbb417d5b50ab6.tar.gz spark-b8a1871953058c67b49b7f8455cbb417d5b50ab6.tar.bz2 spark-b8a1871953058c67b49b7f8455cbb417d5b50ab6.zip | |
SPARK-1053. Don't require SPARK_YARN_APP_JAR
It looks this just requires taking out the checks.
I verified that, with the patch, I was able to run spark-shell through yarn without setting the environment variable.
Author: Sandy Ryza <sandy@cloudera.com>
Closes #553 from sryza/sandy-spark-1053 and squashes the following commits:
b037676 [Sandy Ryza] SPARK-1053. Don't require SPARK_YARN_APP_JAR
Diffstat (limited to 'docs/running-on-yarn.md')
| -rw-r--r-- | docs/running-on-yarn.md | 6 |
1 files changed, 2 insertions, 4 deletions
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index cd4509ede7..ee1d892a3b 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -99,13 +99,12 @@ With this mode, your application is actually run on the remote machine where the ## Launch spark application with yarn-client mode. -With yarn-client mode, the application will be launched locally. Just like running application or spark-shell on Local / Mesos / Standalone mode. The launch method is also the similar with them, just make sure that when you need to specify a master url, use "yarn-client" instead. And you also need to export the env value for SPARK_JAR and SPARK_YARN_APP_JAR +With yarn-client mode, the application will be launched locally. Just like running application or spark-shell on Local / Mesos / Standalone mode. The launch method is also the similar with them, just make sure that when you need to specify a master url, use "yarn-client" instead. And you also need to export the env value for SPARK_JAR. Configuration in yarn-client mode: In order to tune worker core/number/memory etc. You need to export environment variables or add them to the spark configuration file (./conf/spark_env.sh). The following are the list of options. -* `SPARK_YARN_APP_JAR`, Path to your application's JAR file (required) * `SPARK_WORKER_INSTANCES`, Number of workers to start (Default: 2) * `SPARK_WORKER_CORES`, Number of cores for the workers (Default: 1). * `SPARK_WORKER_MEMORY`, Memory per Worker (e.g. 1000M, 2G) (Default: 1G) @@ -118,12 +117,11 @@ In order to tune worker core/number/memory etc. You need to export environment v For example: SPARK_JAR=./assembly/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-assembly-{{site.SPARK_VERSION}}-hadoop2.0.5-alpha.jar \ - SPARK_YARN_APP_JAR=examples/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \ ./bin/run-example org.apache.spark.examples.SparkPi yarn-client +or SPARK_JAR=./assembly/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-assembly-{{site.SPARK_VERSION}}-hadoop2.0.5-alpha.jar \ - SPARK_YARN_APP_JAR=examples/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \ MASTER=yarn-client ./bin/spark-shell |