diff options
| author | Andrew Or <andrewor14@gmail.com> | 2014-03-31 23:01:14 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-03-31 23:01:14 -0700 |
| commit | ada310a9d3d5419e101b24d9b41398f609da1ad3 (patch) | |
| tree | 5ebb735586855143627b14b1073cd6ad07180a4f | |
| parent | 94fe7fd4fa9749cb13e540e4f9caf28de47eaf32 (diff) | |
| download | spark-ada310a9d3d5419e101b24d9b41398f609da1ad3.tar.gz spark-ada310a9d3d5419e101b24d9b41398f609da1ad3.tar.bz2 spark-ada310a9d3d5419e101b24d9b41398f609da1ad3.zip | |
[Hot Fix #42] Persisted RDD disappears on storage page if re-used
If a previously persisted RDD is re-used, its information disappears from the Storage page.
This is because the tasks associated with re-using the RDD do not report the RDD's blocks as updated (which is correct). On stage submit, however, we overwrite any existing information regarding that RDD with a fresh one, whether or not the information for the RDD already exists.
Author: Andrew Or <andrewor14@gmail.com>
Closes #281 from andrewor14/ui-storage-fix and squashes the following commits:
408585a [Andrew Or] Fix storage UI bug
| -rw-r--r-- | core/src/main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala b/core/src/main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala index 4d8b01dbe6..a7b24ff695 100644 --- a/core/src/main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala +++ b/core/src/main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala @@ -84,7 +84,7 @@ private[ui] class BlockManagerListener(storageStatusListener: StorageStatusListe override def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted) = synchronized { val rddInfo = stageSubmitted.stageInfo.rddInfo - _rddInfoMap(rddInfo.id) = rddInfo + _rddInfoMap.getOrElseUpdate(rddInfo.id, rddInfo) } override def onStageCompleted(stageCompleted: SparkListenerStageCompleted) = synchronized { |