aboutsummaryrefslogtreecommitdiff
path: root/docs/configuration.md
diff options
context:
space:
mode:
Diffstat (limited to 'docs/configuration.md')
-rw-r--r--docs/configuration.md61
1 files changed, 61 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md
index 64aa94f622..2c8dea869b 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1008,6 +1008,67 @@ Apart from these, the following properties are also available, and may be useful
</tr>
</table>
+#### Dynamic allocation
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr>
+ <td><code>spark.dynamicAllocation.enabled</code></td>
+ <td>false</td>
+ <td>
+ Whether to use dynamic resource allocation, which scales the number of executors registered
+ with this application up and down based on the workload. Note that this is currently only
+ available on YARN mode. For more detail, see the description
+ <a href="job-scheduling.html#dynamic-resource-allocation">here</a>.
+ <br><br>
+ This requires the following configurations to be set:
+ <code>spark.dynamicAllocation.minExecutors</code>,
+ <code>spark.dynamicAllocation.maxExecutors</code>, and
+ <code>spark.shuffle.service.enabled</code>
+ </td>
+</tr>
+<tr>
+ <td><code>spark.dynamicAllocation.minExecutors</code></td>
+ <td>(none)</td>
+ <td>
+ Lower bound for the number of executors if dynamic allocation is enabled (required).
+ </td>
+</tr>
+<tr>
+ <td><code>spark.dynamicAllocation.maxExecutors</code></td>
+ <td>(none)</td>
+ <td>
+ Upper bound for the number of executors if dynamic allocation is enabled (required).
+ </td>
+</tr>
+<tr>
+ <td><code>spark.dynamicAllocation.schedulerBacklogTimeout</code></td>
+ <td>60</td>
+ <td>
+ If dynamic allocation is enabled and there have been pending tasks backlogged for more than
+ this duration (in seconds), new executors will be requested. For more detail, see this
+ <a href="job-scheduling.html#resource-allocation-policy">description</a>.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.dynamicAllocation.sustainedSchedulerBacklogTimeout</code></td>
+ <td><code>schedulerBacklogTimeout</code></td>
+ <td>
+ Same as <code>spark.dynamicAllocation.schedulerBacklogTimeout</code>, but used only for
+ subsequent executor requests. For more detail, see this
+ <a href="job-scheduling.html#resource-allocation-policy">description</a>.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.dynamicAllocation.executorIdleTimeout</code></td>
+ <td>600</td>
+ <td>
+ If dynamic allocation is enabled and an executor has been idle for more than this duration
+ (in seconds), the executor will be removed. For more detail, see this
+ <a href="job-scheduling.html#resource-allocation-policy">description</a>.
+ </td>
+</tr>
+</table>
+
#### Security
<table class="table">
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>