diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2016-01-06 20:50:31 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-01-06 20:50:31 -0800 |
| commit | 8e19c7663a067d55b32af68d62da42c7cd5d6009 (patch) | |
| tree | 331132f4bc4dcc48d94acda2ff5d456af849ab77 /docs/configuration.md | |
| parent | 6b6d02be0d4e2ce562dddfb391b3302f79de8276 (diff) | |
| download | spark-8e19c7663a067d55b32af68d62da42c7cd5d6009.tar.gz spark-8e19c7663a067d55b32af68d62da42c7cd5d6009.tar.bz2 spark-8e19c7663a067d55b32af68d62da42c7cd5d6009.zip | |
[SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0
This PR removes `spark.cleaner.ttl` and the associated TTL-based metadata cleaning code.
Now that we have the `ContextCleaner` and a timer to trigger periodic GCs, I don't think that `spark.cleaner.ttl` is necessary anymore. The TTL-based cleaning isn't enabled by default, isn't included in our end-to-end tests, and has been a source of user confusion when it is misconfigured. If the TTL is set too low, data which is still being used may be evicted / deleted, leading to hard to diagnose bugs.
For all of these reasons, I think that we should remove this functionality in Spark 2.0. Additional benefits of doing this include marginally reduced memory usage, since we no longer need to store timetsamps in hashmaps, and a handful fewer threads.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #10534 from JoshRosen/remove-ttl-based-cleaning.
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 11 |
1 files changed, 0 insertions, 11 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 7d743d572b..3ffc77dcc6 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -824,17 +824,6 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> - <td><code>spark.cleaner.ttl</code></td> - <td>(infinite)</td> - <td> - Duration (seconds) of how long Spark will remember any metadata (stages generated, tasks - generated, etc.). Periodic cleanups will ensure that metadata older than this duration will be - forgotten. This is useful for running Spark for many hours / days (for example, running 24/7 in - case of Spark Streaming applications). Note that any RDD that persists in memory for more than - this duration will be cleared as well. - </td> -</tr> -<tr> <td><code>spark.executor.cores</code></td> <td>1 in YARN mode, all the available cores on the worker in standalone mode.</td> <td> |