diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2016-01-06 20:50:31 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-01-06 20:50:31 -0800 |
| commit | 8e19c7663a067d55b32af68d62da42c7cd5d6009 (patch) | |
| tree | 331132f4bc4dcc48d94acda2ff5d456af849ab77 /docs/spark-standalone.md | |
| parent | 6b6d02be0d4e2ce562dddfb391b3302f79de8276 (diff) | |
| download | spark-8e19c7663a067d55b32af68d62da42c7cd5d6009.tar.gz spark-8e19c7663a067d55b32af68d62da42c7cd5d6009.tar.bz2 spark-8e19c7663a067d55b32af68d62da42c7cd5d6009.zip | |
[SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0
This PR removes `spark.cleaner.ttl` and the associated TTL-based metadata cleaning code.
Now that we have the `ContextCleaner` and a timer to trigger periodic GCs, I don't think that `spark.cleaner.ttl` is necessary anymore. The TTL-based cleaning isn't enabled by default, isn't included in our end-to-end tests, and has been a source of user confusion when it is misconfigured. If the TTL is set too low, data which is still being used may be evicted / deleted, leading to hard to diagnose bugs.
For all of these reasons, I think that we should remove this functionality in Spark 2.0. Additional benefits of doing this include marginally reduced memory usage, since we no longer need to store timetsamps in hashmaps, and a handful fewer threads.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #10534 from JoshRosen/remove-ttl-based-cleaning.
Diffstat (limited to 'docs/spark-standalone.md')
0 files changed, 0 insertions, 0 deletions