From bb98ecafce196ecc5bc3a1e4cc9264df7b752c6a Mon Sep 17 00:00:00 2001 From: Aaron Davidson Date: Thu, 15 May 2014 21:37:58 -0700 Subject: SPARK-1860: Do not cleanup application work/ directories by default This causes an unrecoverable error for applications that are running for longer than 7 days that have jars added to the SparkContext, as the jars are cleaned up even though the application is still running. Author: Aaron Davidson Closes #800 from aarondav/shitty-defaults and squashes the following commits: a573fbb [Aaron Davidson] SPARK-1860: Do not cleanup application work/ directories by default --- docs/configuration.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) (limited to 'docs') diff --git a/docs/configuration.md b/docs/configuration.md index 2eed96f704..f89040d714 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -390,10 +390,11 @@ Apart from these, the following properties are also available, and may be useful spark.worker.cleanup.enabled - true + false Enable periodic cleanup of worker / application directories. Note that this only affects standalone - mode, as YARN works differently. + mode, as YARN works differently. Applications directories are cleaned up regardless of whether + the application is still running. -- cgit v1.2.3