diff options
| author | Michael Gummelt <mgummelt@mesosphere.io> | 2016-04-26 09:31:53 +0100 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-04-26 09:31:53 +0100 |
| commit | 6a7ba1ff7431281f4c3994a8db70a8fb6eefbf00 (patch) | |
| tree | d94ac91779a8c2ceab42b2f8a9594dd9a2b767d0 | |
| parent | 2a5c930790b4b92674e74f093380d89a9a625552 (diff) | |
| download | spark-6a7ba1ff7431281f4c3994a8db70a8fb6eefbf00.tar.gz spark-6a7ba1ff7431281f4c3994a8db70a8fb6eefbf00.tar.bz2 spark-6a7ba1ff7431281f4c3994a8db70a8fb6eefbf00.zip | |
Fix dynamic allocation docs to address cached data.
## What changes were proposed in this pull request?
Documentation changes
## How was this patch tested?
No tests
Author: Michael Gummelt <mgummelt@mesosphere.io>
Closes #12664 from mgummelt/fix-dynamic-docs.
| -rw-r--r-- | docs/job-scheduling.md | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/docs/job-scheduling.md b/docs/job-scheduling.md index 083c020caa..40b6cd99cc 100644 --- a/docs/job-scheduling.md +++ b/docs/job-scheduling.md @@ -158,8 +158,9 @@ executors will fetch shuffle files from the service instead of from each other. shuffle state written by an executor may continue to be served beyond the executor's lifetime. In addition to writing shuffle files, executors also cache data either on disk or in memory. -When an executor is removed, however, all cached data will no longer be accessible. There is -currently not yet a solution for this in Spark 1.2. In future releases, the cached data may be +When an executor is removed, however, all cached data will no longer be accessible. To mitigate this, +by default executors containing cached data are never removed. You can configure this behavior with +`spark.dynamicAllocation.cachedExecutorIdleTimeout`. In future releases, the cached data may be preserved through an off-heap storage similar in spirit to how shuffle files are preserved through the external shuffle service. |