diff options
| author | WangTaoTheTonic <barneystinson@aliyun.com> | 2015-01-16 09:16:56 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-01-16 09:16:56 -0800 |
| commit | 2be82b1e66cd188456bbf1e5abb13af04d1629d5 (patch) | |
| tree | ba62a9aeb1084739fd75ffd6d991992b70be3af9 /docs/running-on-yarn.md | |
| parent | a79a9f923c47f2ce7da93cf0ecfe2b66fcb9fdd4 (diff) | |
| download | spark-2be82b1e66cd188456bbf1e5abb13af04d1629d5.tar.gz spark-2be82b1e66cd188456bbf1e5abb13af04d1629d5.tar.bz2 spark-2be82b1e66cd188456bbf1e5abb13af04d1629d5.zip | |
[SPARK-1507][YARN]specify # cores for ApplicationMaster
Based on top of changes in https://github.com/apache/spark/pull/3806.
https://issues.apache.org/jira/browse/SPARK-1507
`--driver-cores` and `spark.driver.cores` for all cluster modes and `spark.yarn.am.cores` for yarn client mode.
Author: WangTaoTheTonic <barneystinson@aliyun.com>
Author: WangTao <barneystinson@aliyun.com>
Closes #4018 from WangTaoTheTonic/SPARK-1507 and squashes the following commits:
01419d3 [WangTaoTheTonic] amend the args name
b255795 [WangTaoTheTonic] indet thing
d86557c [WangTaoTheTonic] some comments amend
43c9392 [WangTao] fix compile error
b39a100 [WangTao] specify # cores for ApplicationMaster
Diffstat (limited to 'docs/running-on-yarn.md')
| -rw-r--r-- | docs/running-on-yarn.md | 17 |
1 files changed, 17 insertions, 0 deletions
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 4f273098c5..68ab127bcf 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -30,6 +30,23 @@ Most of the configs are the same for Spark on YARN as for other deployment modes </td> </tr> <tr> + <td><code>spark.driver.cores</code></td> + <td>1</td> + <td> + Number of cores used by the driver in YARN cluster mode. + Since the driver is run in the same JVM as the YARN Application Master in cluster mode, this also controls the cores used by the YARN AM. + In client mode, use <code>spark.yarn.am.cores</code> to control the number of cores used by the YARN AM instead. + </td> +</tr> +<tr> + <td><code>spark.yarn.am.cores</code></td> + <td>1</td> + <td> + Number of cores to use for the YARN Application Master in client mode. + In cluster mode, use <code>spark.driver.cores</code> instead. + </td> +</tr> +<tr> <td><code>spark.yarn.am.waitTime</code></td> <td>100000</td> <td> |