diff options
| author | Matei Zaharia <matei@databricks.com> | 2014-02-24 13:14:56 -0800 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-02-24 13:14:56 -0800 |
| commit | 0187cef0f284e6cb22cb3986c327c43304daf57d (patch) | |
| tree | de0c21e8ffd3dad5567fed708d145e0f7ea9df64 | |
| parent | cd32d5e4dee1291e4509e5965322b7ffe620b1f3 (diff) | |
| download | spark-0187cef0f284e6cb22cb3986c327c43304daf57d.tar.gz spark-0187cef0f284e6cb22cb3986c327c43304daf57d.tar.bz2 spark-0187cef0f284e6cb22cb3986c327c43304daf57d.zip | |
Fix removal from shuffleToMapStage to search for a key-value pair with
our stage instead of using our shuffleID.
| -rw-r--r-- | core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala index 789d5e6699..dc5b25d845 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala @@ -381,8 +381,8 @@ class DAGScheduler( running -= stage } stageToInfos -= stage - for (shuffleDep <- stage.shuffleDep) { - shuffleToMapStage.remove(shuffleDep.shuffleId) + for ((k, v) <- shuffleToMapStage.find(_._2 == stage)) { + shuffleToMapStage.remove(k) } if (pendingTasks.contains(stage) && !pendingTasks(stage).isEmpty) { logDebug("Removing pending status for stage %d".format(stageId)) |