diff options
| author | Kousuke Saruta <sarutak@oss.nttdata.co.jp> | 2015-01-08 13:43:09 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-01-08 13:43:20 -0800 |
| commit | 71471bd79579e6cbbced1f4ccea6fcecc9bfc656 (patch) | |
| tree | 872c5bd4587751c3a3ab98c69195e9e476162241 /core/src/main | |
| parent | 755f9cc403357da9e5673b5035ebec100b1d83e6 (diff) | |
| download | spark-71471bd79579e6cbbced1f4ccea6fcecc9bfc656.tar.gz spark-71471bd79579e6cbbced1f4ccea6fcecc9bfc656.tar.bz2 spark-71471bd79579e6cbbced1f4ccea6fcecc9bfc656.zip | |
[SPARK-4973][CORE] Local directory in the driver of client-mode continues remaining even if application finished when external shuffle is enabled
When we enables external shuffle service, local directories in the driver of client-mode continue remaining even if application has finished.
I think local directories for drivers should be deleted.
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Closes #3811 from sarutak/SPARK-4973 and squashes the following commits:
ad944ab [Kousuke Saruta] Fixed DiskBlockManager to cleanup local directory if it's the driver
43770da [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-4973
88feecd [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-4973
d99718e [Kousuke Saruta] Fixed SparkSubmit.scala and DiskBlockManager.scala in order to delete local directories of the driver of local-mode when external shuffle service is enabled
(cherry picked from commit a00af6bec57b8df8b286aaa5897232475aef441c)
Signed-off-by: Andrew Or <andrew@databricks.com>
Diffstat (limited to 'core/src/main')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala b/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala index 58fba54710..d79ed76542 100644 --- a/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala +++ b/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala @@ -164,7 +164,7 @@ private[spark] class DiskBlockManager(blockManager: BlockManager, conf: SparkCon /** Cleanup local dirs and stop shuffle sender. */ private[spark] def stop() { // Only perform cleanup if an external service is not serving our shuffle files. - if (!blockManager.externalShuffleServiceEnabled) { + if (!blockManager.externalShuffleServiceEnabled || blockManager.blockManagerId.isDriver) { localDirs.foreach { localDir => if (localDir.isDirectory() && localDir.exists()) { try { |