diff options
author | Andrew Or <andrewor14@gmail.com> | 2014-06-17 01:28:22 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-06-17 01:28:37 -0700 |
commit | 3d4fa2dab0ade3b7948497a3336fd3e238f93507 (patch) | |
tree | ff2f14a88a12f05ff8ef88a983182d262780a3da /core/src | |
parent | 6b5f64aafcfe023851039f8caf7ba71e7d56d26e (diff) | |
download | spark-3d4fa2dab0ade3b7948497a3336fd3e238f93507.tar.gz spark-3d4fa2dab0ade3b7948497a3336fd3e238f93507.tar.bz2 spark-3d4fa2dab0ade3b7948497a3336fd3e238f93507.zip |
[SPARK-2144] ExecutorsPage reports incorrect # of RDD blocks
This is reproducible whenever we drop a block because of memory pressure.
This is because StorageStatusListener actually never removes anything from the block maps of its StorageStatuses. Instead, when a block is dropped, it sets the block's storage level to `StorageLevel.NONE`, when it should just remove it from the map.
This PR includes this simple fix.
Author: Andrew Or <andrewor14@gmail.com>
Closes #1080 from andrewor14/ui-blocks and squashes the following commits:
fcf9f1a [Andrew Or] Remove BlockStatus if it is no longer cached
(cherry picked from commit 09deb3eee090eb8ec1d9a0cd90825699748e3ffc)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Diffstat (limited to 'core/src')
-rw-r--r-- | core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala | 6 |
1 files changed, 5 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala index a6e6627d54..c694fc8c34 100644 --- a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala +++ b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala @@ -37,7 +37,11 @@ class StorageStatusListener extends SparkListener { val filteredStatus = storageStatusList.find(_.blockManagerId.executorId == execId) filteredStatus.foreach { storageStatus => updatedBlocks.foreach { case (blockId, updatedStatus) => - storageStatus.blocks(blockId) = updatedStatus + if (updatedStatus.storageLevel == StorageLevel.NONE) { + storageStatus.blocks.remove(blockId) + } else { + storageStatus.blocks(blockId) = updatedStatus + } } } } |