diff options
| author | Aaron Davidson <aaron@databricks.com> | 2014-05-15 21:37:58 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-05-15 21:38:25 -0700 |
| commit | 54414716ba9d3f02cfcaccf292d6254783617f78 (patch) | |
| tree | d1b02becebb337f7b5453f9b6903dae0e62e8183 /docs | |
| parent | a2742d8506463dd0c7bbab06abcd68a0ae44c8e5 (diff) | |
| download | spark-54414716ba9d3f02cfcaccf292d6254783617f78.tar.gz spark-54414716ba9d3f02cfcaccf292d6254783617f78.tar.bz2 spark-54414716ba9d3f02cfcaccf292d6254783617f78.zip | |
SPARK-1860: Do not cleanup application work/ directories by default
This causes an unrecoverable error for applications that are running for longer
than 7 days that have jars added to the SparkContext, as the jars are cleaned up
even though the application is still running.
Author: Aaron Davidson <aaron@databricks.com>
Closes #800 from aarondav/shitty-defaults and squashes the following commits:
a573fbb [Aaron Davidson] SPARK-1860: Do not cleanup application work/ directories by default
(cherry picked from commit bb98ecafce196ecc5bc3a1e4cc9264df7b752c6a)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configuration.md | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 2eed96f704..f89040d714 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -390,10 +390,11 @@ Apart from these, the following properties are also available, and may be useful </tr> <tr> <td>spark.worker.cleanup.enabled</td> - <td>true</td> + <td>false</td> <td> Enable periodic cleanup of worker / application directories. Note that this only affects standalone - mode, as YARN works differently. + mode, as YARN works differently. Applications directories are cleaned up regardless of whether + the application is still running. </td> </tr> <tr> |