diff options
author | Zhongshuai Pei <peizhongshuai@huawei.com> | 2015-11-25 10:37:34 -0800 |
---|---|---|
committer | Reynold Xin <rxin@databricks.com> | 2015-11-25 10:37:34 -0800 |
commit | 6b781576a15d8d5c5fbed8bef1c5bda95b3d44ac (patch) | |
tree | a2389ba5672829d0202bf4d38d318cfb0e02280d /core/src | |
parent | faabdfa2bd416ae514961535f1953e8e9e8b1f3f (diff) | |
download | spark-6b781576a15d8d5c5fbed8bef1c5bda95b3d44ac.tar.gz spark-6b781576a15d8d5c5fbed8bef1c5bda95b3d44ac.tar.bz2 spark-6b781576a15d8d5c5fbed8bef1c5bda95b3d44ac.zip |
[SPARK-11974][CORE] Not all the temp dirs had been deleted when the JVM exits
deleting the temp dir like that
```
scala> import scala.collection.mutable
import scala.collection.mutable
scala> val a = mutable.Set(1,2,3,4,7,0,8,98,9)
a: scala.collection.mutable.Set[Int] = Set(0, 9, 1, 2, 3, 7, 4, 8, 98)
scala> a.foreach(x => {a.remove(x) })
scala> a.foreach(println(_))
98
```
You may not modify a collection while traversing or iterating over it.This can not delete all element of the collection
Author: Zhongshuai Pei <peizhongshuai@huawei.com>
Closes #9951 from DoingDone9/Bug_RemainDir.
Diffstat (limited to 'core/src')
-rw-r--r-- | core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala b/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala index db4a8b304e..4012dca3ec 100644 --- a/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala +++ b/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala @@ -57,7 +57,9 @@ private[spark] object ShutdownHookManager extends Logging { // Add a shutdown hook to delete the temp dirs when the JVM exits addShutdownHook(TEMP_DIR_SHUTDOWN_PRIORITY) { () => logInfo("Shutdown hook called") - shutdownDeletePaths.foreach { dirPath => + // we need to materialize the paths to delete because deleteRecursively removes items from + // shutdownDeletePaths as we are traversing through it. + shutdownDeletePaths.toArray.foreach { dirPath => try { logInfo("Deleting directory " + dirPath) Utils.deleteRecursively(new File(dirPath)) |