diff options
| author | Hari Shreedharan <hshreedharan@apache.org> | 2015-06-06 21:13:26 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-06-06 21:13:26 -0700 |
| commit | 3285a51121397bfd2e62dbee8e1f0fa7c72512a7 (patch) | |
| tree | fcc9452d5fbc94ce40d47d831c7beacc40812247 /docs/configuration.md | |
| parent | ed2cc3ee890694ca0c1fa0bbc7186c8b80da3fab (diff) | |
| download | spark-3285a51121397bfd2e62dbee8e1f0fa7c72512a7.tar.gz spark-3285a51121397bfd2e62dbee8e1f0fa7c72512a7.tar.bz2 spark-3285a51121397bfd2e62dbee8e1f0fa7c72512a7.zip | |
[SPARK-7955] [CORE] Ensure executors with cached RDD blocks are not re…
…moved if dynamic allocation is enabled.
This is a work in progress. This patch ensures that an executor that has cached RDD blocks are not removed,
but makes no attempt to find another executor to remove. This is meant to get some feedback on the current
approach, and if it makes sense then I will look at choosing another executor to remove. No testing has been done either.
Author: Hari Shreedharan <hshreedharan@apache.org>
Closes #6508 from harishreedharan/dymanic-caching and squashes the following commits:
dddf1eb [Hari Shreedharan] Minor configuration description update.
10130e2 [Hari Shreedharan] Fix compile issue.
5417b53 [Hari Shreedharan] Add documentation for new config. Remove block from cachedBlocks when it is dropped.
875916a [Hari Shreedharan] Make some code more readable.
39940ca [Hari Shreedharan] Handle the case where the executor has not yet registered.
90ad711 [Hari Shreedharan] Remove unused imports and unused methods.
063985c [Hari Shreedharan] Send correct message instead of recursively calling same method.
ec2fd7e [Hari Shreedharan] Add file missed in last commit
5d10fad [Hari Shreedharan] Update cached blocks status using local info, rather than doing an RPC.
193af4c [Hari Shreedharan] WIP. Use local state rather than via RPC.
ae932ff [Hari Shreedharan] Fix config param name.
272969d [Hari Shreedharan] Fix seconds to millis bug.
5a1993f [Hari Shreedharan] Add timeout for cache executors. Ignore broadcast blocks while checking if there are cached blocks.
57fefc2 [Hari Shreedharan] [SPARK-7955][Core] Ensure executors with cached RDD blocks are not removed if dynamic allocation is enabled.
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 9 |
1 files changed, 9 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 3a48da4592..9667cebe0b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1202,6 +1202,15 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.dynamicAllocation.cachedExecutorIdleTimeout</code></td> + <td>2 * executorIdleTimeout</td> + <td> + If dynamic allocation is enabled and an executor which has cached data blocks has been idle for more than this duration, + the executor will be removed. For more details, see this + <a href="job-scheduling.html#resource-allocation-policy">description</a>. + </td> +</tr> +<tr> <td><code>spark.dynamicAllocation.initialExecutors</code></td> <td><code>spark.dynamicAllocation.minExecutors</code></td> <td> |