diff options
Diffstat (limited to 'docs')
-rw-r--r-- | docs/configuration.md | 15 | ||||
-rw-r--r-- | docs/running-on-yarn.md | 17 |
2 files changed, 28 insertions, 4 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 673cdb371a..efbab40853 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -102,11 +102,10 @@ of the most common options to set are: </td> </tr> <tr> - <td><code>spark.executor.memory</code></td> - <td>512m</td> + <td><code>spark.driver.cores</code></td> + <td>1</td> <td> - Amount of memory to use per executor process, in the same format as JVM memory strings - (e.g. <code>512m</code>, <code>2g</code>). + Number of cores to use for the driver process, only in cluster mode. </td> </tr> <tr> @@ -118,6 +117,14 @@ of the most common options to set are: </td> </tr> <tr> + <td><code>spark.executor.memory</code></td> + <td>512m</td> + <td> + Amount of memory to use per executor process, in the same format as JVM memory strings + (e.g. <code>512m</code>, <code>2g</code>). + </td> +</tr> +<tr> <td><code>spark.driver.maxResultSize</code></td> <td>1g</td> <td> diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 4f273098c5..68ab127bcf 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -30,6 +30,23 @@ Most of the configs are the same for Spark on YARN as for other deployment modes </td> </tr> <tr> + <td><code>spark.driver.cores</code></td> + <td>1</td> + <td> + Number of cores used by the driver in YARN cluster mode. + Since the driver is run in the same JVM as the YARN Application Master in cluster mode, this also controls the cores used by the YARN AM. + In client mode, use <code>spark.yarn.am.cores</code> to control the number of cores used by the YARN AM instead. + </td> +</tr> +<tr> + <td><code>spark.yarn.am.cores</code></td> + <td>1</td> + <td> + Number of cores to use for the YARN Application Master in client mode. + In cluster mode, use <code>spark.driver.cores</code> instead. + </td> +</tr> +<tr> <td><code>spark.yarn.am.waitTime</code></td> <td>100000</td> <td> |