aboutsummaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
Diffstat (limited to 'docs')
-rw-r--r--docs/configuration.md15
-rw-r--r--docs/running-on-yarn.md17
2 files changed, 28 insertions, 4 deletions
diff --git a/docs/configuration.md b/docs/configuration.md
index 673cdb371a..efbab40853 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -102,11 +102,10 @@ of the most common options to set are:
</td>
</tr>
<tr>
- <td><code>spark.executor.memory</code></td>
- <td>512m</td>
+ <td><code>spark.driver.cores</code></td>
+ <td>1</td>
<td>
- Amount of memory to use per executor process, in the same format as JVM memory strings
- (e.g. <code>512m</code>, <code>2g</code>).
+ Number of cores to use for the driver process, only in cluster mode.
</td>
</tr>
<tr>
@@ -118,6 +117,14 @@ of the most common options to set are:
</td>
</tr>
<tr>
+ <td><code>spark.executor.memory</code></td>
+ <td>512m</td>
+ <td>
+ Amount of memory to use per executor process, in the same format as JVM memory strings
+ (e.g. <code>512m</code>, <code>2g</code>).
+ </td>
+</tr>
+<tr>
<td><code>spark.driver.maxResultSize</code></td>
<td>1g</td>
<td>
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index 4f273098c5..68ab127bcf 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -30,6 +30,23 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
</td>
</tr>
<tr>
+ <td><code>spark.driver.cores</code></td>
+ <td>1</td>
+ <td>
+ Number of cores used by the driver in YARN cluster mode.
+ Since the driver is run in the same JVM as the YARN Application Master in cluster mode, this also controls the cores used by the YARN AM.
+ In client mode, use <code>spark.yarn.am.cores</code> to control the number of cores used by the YARN AM instead.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.yarn.am.cores</code></td>
+ <td>1</td>
+ <td>
+ Number of cores to use for the YARN Application Master in client mode.
+ In cluster mode, use <code>spark.driver.cores</code> instead.
+ </td>
+</tr>
+<tr>
<td><code>spark.yarn.am.waitTime</code></td>
<td>100000</td>
<td>