diff options
Diffstat (limited to 'docs/configuration.md')
-rw-r--r-- | docs/configuration.md | 61 |
1 files changed, 61 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 64aa94f622..2c8dea869b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1008,6 +1008,67 @@ Apart from these, the following properties are also available, and may be useful </tr> </table> +#### Dynamic allocation +<table class="table"> +<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr> +<tr> + <td><code>spark.dynamicAllocation.enabled</code></td> + <td>false</td> + <td> + Whether to use dynamic resource allocation, which scales the number of executors registered + with this application up and down based on the workload. Note that this is currently only + available on YARN mode. For more detail, see the description + <a href="job-scheduling.html#dynamic-resource-allocation">here</a>. + <br><br> + This requires the following configurations to be set: + <code>spark.dynamicAllocation.minExecutors</code>, + <code>spark.dynamicAllocation.maxExecutors</code>, and + <code>spark.shuffle.service.enabled</code> + </td> +</tr> +<tr> + <td><code>spark.dynamicAllocation.minExecutors</code></td> + <td>(none)</td> + <td> + Lower bound for the number of executors if dynamic allocation is enabled (required). + </td> +</tr> +<tr> + <td><code>spark.dynamicAllocation.maxExecutors</code></td> + <td>(none)</td> + <td> + Upper bound for the number of executors if dynamic allocation is enabled (required). + </td> +</tr> +<tr> + <td><code>spark.dynamicAllocation.schedulerBacklogTimeout</code></td> + <td>60</td> + <td> + If dynamic allocation is enabled and there have been pending tasks backlogged for more than + this duration (in seconds), new executors will be requested. For more detail, see this + <a href="job-scheduling.html#resource-allocation-policy">description</a>. + </td> +</tr> +<tr> + <td><code>spark.dynamicAllocation.sustainedSchedulerBacklogTimeout</code></td> + <td><code>schedulerBacklogTimeout</code></td> + <td> + Same as <code>spark.dynamicAllocation.schedulerBacklogTimeout</code>, but used only for + subsequent executor requests. For more detail, see this + <a href="job-scheduling.html#resource-allocation-policy">description</a>. + </td> +</tr> +<tr> + <td><code>spark.dynamicAllocation.executorIdleTimeout</code></td> + <td>600</td> + <td> + If dynamic allocation is enabled and an executor has been idle for more than this duration + (in seconds), the executor will be removed. For more detail, see this + <a href="job-scheduling.html#resource-allocation-policy">description</a>. + </td> +</tr> +</table> + #### Security <table class="table"> <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr> |