diff options
| author | Sandy Ryza <sandy@cloudera.com> | 2015-02-02 12:27:08 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-02-02 12:27:08 -0800 |
| commit | b2047b55c5fc85de6b63276d8ab9610d2496e08b (patch) | |
| tree | b62017de84f2893af77a00714988e72b75c257a8 /docs | |
| parent | c081b21b1fe4fbad845088c4144da0bd2a8d89dc (diff) | |
| download | spark-b2047b55c5fc85de6b63276d8ab9610d2496e08b.tar.gz spark-b2047b55c5fc85de6b63276d8ab9610d2496e08b.tar.bz2 spark-b2047b55c5fc85de6b63276d8ab9610d2496e08b.zip | |
SPARK-4585. Spark dynamic executor allocation should use minExecutors as...
... initial number
Author: Sandy Ryza <sandy@cloudera.com>
Closes #4051 from sryza/sandy-spark-4585 and squashes the following commits:
d1dd039 [Sandy Ryza] Add spark.dynamicAllocation.initialNumExecutors and make min and max not required
b7c59dc [Sandy Ryza] SPARK-4585. Spark dynamic executor allocation should use minExecutors as initial number
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configuration.md | 20 | ||||
| -rw-r--r-- | docs/job-scheduling.md | 9 |
2 files changed, 18 insertions, 11 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index e4e4b8d516..08c6befaf3 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1098,24 +1098,32 @@ Apart from these, the following properties are also available, and may be useful available on YARN mode. For more detail, see the description <a href="job-scheduling.html#dynamic-resource-allocation">here</a>. <br><br> - This requires the following configurations to be set: + This requires <code>spark.shuffle.service.enabled</code> to be set. + The following configurations are also relevant: <code>spark.dynamicAllocation.minExecutors</code>, <code>spark.dynamicAllocation.maxExecutors</code>, and - <code>spark.shuffle.service.enabled</code> + <code>spark.dynamicAllocation.initialExecutors</code> </td> </tr> <tr> <td><code>spark.dynamicAllocation.minExecutors</code></td> - <td>(none)</td> + <td>0</td> <td> - Lower bound for the number of executors if dynamic allocation is enabled (required). + Lower bound for the number of executors if dynamic allocation is enabled. </td> </tr> <tr> <td><code>spark.dynamicAllocation.maxExecutors</code></td> - <td>(none)</td> + <td>Integer.MAX_VALUE</td> + <td> + Upper bound for the number of executors if dynamic allocation is enabled. + </td> +</tr> +<tr> + <td><code>spark.dynamicAllocation.maxExecutors</code></td> + <td><code>spark.dynamicAllocation.minExecutors</code></td> <td> - Upper bound for the number of executors if dynamic allocation is enabled (required). + Initial number of executors to run if dynamic allocation is enabled. </td> </tr> <tr> diff --git a/docs/job-scheduling.md b/docs/job-scheduling.md index a5425eb355..5295e351dd 100644 --- a/docs/job-scheduling.md +++ b/docs/job-scheduling.md @@ -77,11 +77,10 @@ scheduling while sharing cluster resources efficiently. ### Configuration and Setup All configurations used by this feature live under the `spark.dynamicAllocation.*` namespace. -To enable this feature, your application must set `spark.dynamicAllocation.enabled` to `true` and -provide lower and upper bounds for the number of executors through -`spark.dynamicAllocation.minExecutors` and `spark.dynamicAllocation.maxExecutors`. Other relevant -configurations are described on the [configurations page](configuration.html#dynamic-allocation) -and in the subsequent sections in detail. +To enable this feature, your application must set `spark.dynamicAllocation.enabled` to `true`. +Other relevant configurations are described on the +[configurations page](configuration.html#dynamic-allocation) and in the subsequent sections in +detail. Additionally, your application must use an external shuffle service. The purpose of the service is to preserve the shuffle files written by executors so the executors can be safely removed (more |