diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2016-07-14 09:42:32 -0500 |
|---|---|---|
| committer | Tom Graves <tgraves@yahoo-inc.com> | 2016-07-14 09:42:32 -0500 |
| commit | b7b5e17876f65c6644505c356f1a0db24ce1d142 (patch) | |
| tree | e4e6ae132a696260191ecb7e547457d6bbf5e4bf /docs/job-scheduling.md | |
| parent | c4bc2ed844ea045d2e8218154690b5b2b023f1e5 (diff) | |
| download | spark-b7b5e17876f65c6644505c356f1a0db24ce1d142.tar.gz spark-b7b5e17876f65c6644505c356f1a0db24ce1d142.tar.bz2 spark-b7b5e17876f65c6644505c356f1a0db24ce1d142.zip | |
[SPARK-16505][YARN] Optionally propagate error during shuffle service startup.
This prevents the NM from starting when something is wrong, which would
lead to later errors which are confusing and harder to debug.
Added a unit test to verify startup fails if something is wrong.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #14162 from vanzin/SPARK-16505.
Diffstat (limited to 'docs/job-scheduling.md')
| -rw-r--r-- | docs/job-scheduling.md | 13 |
1 files changed, 1 insertions, 12 deletions
diff --git a/docs/job-scheduling.md b/docs/job-scheduling.md index 40b6cd99cc..807944f20a 100644 --- a/docs/job-scheduling.md +++ b/docs/job-scheduling.md @@ -83,18 +83,7 @@ In Mesos coarse-grained mode, run `$SPARK_HOME/sbin/start-mesos-shuffle-service. slave nodes with `spark.shuffle.service.enabled` set to `true`. For instance, you may do so through Marathon. -In YARN mode, start the shuffle service on each `NodeManager` as follows: - -1. Build Spark with the [YARN profile](building-spark.html). Skip this step if you are using a -pre-packaged distribution. -2. Locate the `spark-<version>-yarn-shuffle.jar`. This should be under -`$SPARK_HOME/common/network-yarn/target/scala-<version>` if you are building Spark yourself, and under -`lib` if you are using a distribution. -2. Add this jar to the classpath of all `NodeManager`s in your cluster. -3. In the `yarn-site.xml` on each node, add `spark_shuffle` to `yarn.nodemanager.aux-services`, -then set `yarn.nodemanager.aux-services.spark_shuffle.class` to -`org.apache.spark.network.yarn.YarnShuffleService`. -4. Restart all `NodeManager`s in your cluster. +In YARN mode, follow the instructions [here](running-on-yarn.html#configuring-the-external-shuffle-service). All other relevant configurations are optional and under the `spark.dynamicAllocation.*` and `spark.shuffle.service.*` namespaces. For more detail, see the |