diff options
| author | Kousuke Saruta <sarutak@oss.nttdata.co.jp> | 2015-01-08 13:43:09 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-01-08 13:43:09 -0800 |
| commit | a00af6bec57b8df8b286aaa5897232475aef441c (patch) | |
| tree | 794fd2cb5e32ac87a823562146fef8b521c9ee93 | |
| parent | 72df5a301e706d9384f3a1c17b2c58b017632b1f (diff) | |
| download | spark-a00af6bec57b8df8b286aaa5897232475aef441c.tar.gz spark-a00af6bec57b8df8b286aaa5897232475aef441c.tar.bz2 spark-a00af6bec57b8df8b286aaa5897232475aef441c.zip | |
[SPARK-4973][CORE] Local directory in the driver of client-mode continues remaining even if application finished when external shuffle is enabled
When we enables external shuffle service, local directories in the driver of client-mode continue remaining even if application has finished.
I think local directories for drivers should be deleted.
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Closes #3811 from sarutak/SPARK-4973 and squashes the following commits:
ad944ab [Kousuke Saruta] Fixed DiskBlockManager to cleanup local directory if it's the driver
43770da [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-4973
88feecd [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-4973
d99718e [Kousuke Saruta] Fixed SparkSubmit.scala and DiskBlockManager.scala in order to delete local directories of the driver of local-mode when external shuffle service is enabled
| -rw-r--r-- | core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala b/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala index bb2ae9f3f4..af05eb3ca6 100644 --- a/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala +++ b/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala @@ -166,7 +166,7 @@ private[spark] class DiskBlockManager(blockManager: BlockManager, conf: SparkCon /** Cleanup local dirs and stop shuffle sender. */ private[spark] def stop() { // Only perform cleanup if an external service is not serving our shuffle files. - if (!blockManager.externalShuffleServiceEnabled) { + if (!blockManager.externalShuffleServiceEnabled || blockManager.blockManagerId.isDriver) { localDirs.foreach { localDir => if (localDir.isDirectory() && localDir.exists()) { try { |