diff options
| author | Raymond Liu <raymond.liu@intel.com> | 2014-06-18 10:57:45 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@apache.org> | 2014-06-18 10:57:45 -0700 |
| commit | 5ad5e3486aa4d13b0991de13f5f17d9897dd2753 (patch) | |
| tree | 05a91579f4a2d2602310441257483b856ef8dcc8 /core | |
| parent | 587d32012ceeec1e80cec1878312f164cdb76ec8 (diff) | |
| download | spark-5ad5e3486aa4d13b0991de13f5f17d9897dd2753.tar.gz spark-5ad5e3486aa4d13b0991de13f5f17d9897dd2753.tar.bz2 spark-5ad5e3486aa4d13b0991de13f5f17d9897dd2753.zip | |
[SPARK-2162] Double check in doGetLocal to avoid read on removed block.
other wise, it will either read in vain in memory level case, or throw exception in disk level case when it believe the block is there while actually it had been removed.
Author: Raymond Liu <raymond.liu@intel.com>
Closes #1103 from colorant/bm and squashes the following commits:
daac114 [Raymond Liu] Address comments
d1ea287 [Raymond Liu] Double check in doGetLocal to avoid read on removed block.
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/storage/BlockManager.scala | 7 |
1 files changed, 7 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala index f52bc70751..373987c122 100644 --- a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala +++ b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala @@ -363,6 +363,13 @@ private[spark] class BlockManager( val info = blockInfo.get(blockId).orNull if (info != null) { info.synchronized { + // Double check to make sure the block is still there, since removeBlock + // method also synchronizes on BlockInfo object, so the block might have + // been removed when we actually come here. + if (blockInfo.get(blockId).isEmpty) { + logDebug(s"Block $blockId had been removed") + return None + } // If another thread is writing the block, wait for it to become ready. if (!info.waitForReady()) { |