diff options
author | Sean Owen <sowen@cloudera.com> | 2015-08-06 19:29:42 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2015-08-06 19:29:42 +0100 |
commit | 0d7aac99da660cc42eb5a9be8e262bd9bd8a770f (patch) | |
tree | f5b956d776a2aca9a2c7dd3f28b9e08009293e9b | |
parent | cdd53b762bf358616b313e3334b5f6945caf9ab1 (diff) | |
download | spark-0d7aac99da660cc42eb5a9be8e262bd9bd8a770f.tar.gz spark-0d7aac99da660cc42eb5a9be8e262bd9bd8a770f.tar.bz2 spark-0d7aac99da660cc42eb5a9be8e262bd9bd8a770f.zip |
[SPARK-9641] [DOCS] spark.shuffle.service.port is not documented
Document spark.shuffle.service.{enabled,port}
CC sryza tgravescs
This is pretty minimal; is there more to say here about the service?
Author: Sean Owen <sowen@cloudera.com>
Closes #7991 from srowen/SPARK-9641 and squashes the following commits:
3bb946e [Sean Owen] Add link to docs for setup and config of external shuffle service
2302e01 [Sean Owen] Document spark.shuffle.service.{enabled,port}
-rw-r--r-- | docs/configuration.md | 19 |
1 files changed, 19 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 24b606356a..c60dd16839 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -474,6 +474,25 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.shuffle.service.enabled</code></td> + <td>false</td> + <td> + Enables the external shuffle service. This service preserves the shuffle files written by + executors so the executors can be safely removed. This must be enabled if + <code>spark.dynamicAllocation.enabled</code> is "true". The external shuffle service + must be set up in order to enable it. See + <a href="job-scheduling.html#configuration-and-setup">dynamic allocation + configuration and setup documentation</a> for more information. + </td> +</tr> +<tr> + <td><code>spark.shuffle.service.port</code></td> + <td>7337</td> + <td> + Port on which the external shuffle service will run. + </td> +</tr> +<tr> <td><code>spark.shuffle.sort.bypassMergeThreshold</code></td> <td>200</td> <td> |