diff options
author | Andrew Or <andrewor14@gmail.com> | 2014-06-17 01:28:22 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-06-17 01:28:22 -0700 |
commit | 09deb3eee090eb8ec1d9a0cd90825699748e3ffc (patch) | |
tree | 3c71c7aebd65f951ae46768c3802cdf609511b3e | |
parent | 23a12ce20c55653b08b16e6159ab31d2ca88acf1 (diff) | |
download | spark-09deb3eee090eb8ec1d9a0cd90825699748e3ffc.tar.gz spark-09deb3eee090eb8ec1d9a0cd90825699748e3ffc.tar.bz2 spark-09deb3eee090eb8ec1d9a0cd90825699748e3ffc.zip |
[SPARK-2144] ExecutorsPage reports incorrect # of RDD blocks
This is reproducible whenever we drop a block because of memory pressure.
This is because StorageStatusListener actually never removes anything from the block maps of its StorageStatuses. Instead, when a block is dropped, it sets the block's storage level to `StorageLevel.NONE`, when it should just remove it from the map.
This PR includes this simple fix.
Author: Andrew Or <andrewor14@gmail.com>
Closes #1080 from andrewor14/ui-blocks and squashes the following commits:
fcf9f1a [Andrew Or] Remove BlockStatus if it is no longer cached
-rw-r--r-- | core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala | 6 |
1 files changed, 5 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala index a6e6627d54..c694fc8c34 100644 --- a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala +++ b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala @@ -37,7 +37,11 @@ class StorageStatusListener extends SparkListener { val filteredStatus = storageStatusList.find(_.blockManagerId.executorId == execId) filteredStatus.foreach { storageStatus => updatedBlocks.foreach { case (blockId, updatedStatus) => - storageStatus.blocks(blockId) = updatedStatus + if (updatedStatus.storageLevel == StorageLevel.NONE) { + storageStatus.blocks.remove(blockId) + } else { + storageStatus.blocks(blockId) = updatedStatus + } } } } |