diff options
| author | Holden Karau <holden@us.ibm.com> | 2016-05-03 00:18:10 -0700 |
|---|---|---|
| committer | Xiangrui Meng <meng@databricks.com> | 2016-05-03 00:18:10 -0700 |
| commit | f10ae4b1e169495af11b8e8123c60dd96174477e (patch) | |
| tree | 323357bac8e3b933780625c92292a53c7043e17a /core | |
| parent | d8f528ceb61e3c2ac7ac97cd8147dafbb625932f (diff) | |
| download | spark-f10ae4b1e169495af11b8e8123c60dd96174477e.tar.gz spark-f10ae4b1e169495af11b8e8123c60dd96174477e.tar.bz2 spark-f10ae4b1e169495af11b8e8123c60dd96174477e.zip | |
[SPARK-6717][ML] Clear shuffle files after checkpointing in ALS
## What changes were proposed in this pull request?
When ALS is run with a checkpoint interval, during the checkpoint materialize the current state and cleanup the previous shuffles (non-blocking).
## How was this patch tested?
Existing ALS unit tests, new ALS checkpoint cleanup unit tests added & shuffle files checked after ALS w/checkpointing run.
Author: Holden Karau <holden@us.ibm.com>
Author: Holden Karau <holden@pigscanfly.ca>
Closes #11919 from holdenk/SPARK-6717-clear-shuffle-files-after-checkpointing-in-ALS.
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/ContextCleaner.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/ContextCleaner.scala b/core/src/main/scala/org/apache/spark/ContextCleaner.scala index a51338c017..c895fb3206 100644 --- a/core/src/main/scala/org/apache/spark/ContextCleaner.scala +++ b/core/src/main/scala/org/apache/spark/ContextCleaner.scala @@ -212,7 +212,7 @@ private[spark] class ContextCleaner(sc: SparkContext) extends Logging { } } - /** Perform shuffle cleanup, asynchronously. */ + /** Perform shuffle cleanup. */ def doCleanupShuffle(shuffleId: Int, blocking: Boolean): Unit = { try { logDebug("Cleaning shuffle " + shuffleId) |