diff options
| author | Ernest <earneyzxl@gmail.com> | 2016-03-23 10:29:36 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2016-03-23 10:29:36 -0700 |
| commit | 48ee16d8012602c75d50aa2a85e26b7de3c48944 (patch) | |
| tree | 902eecf48d82ba1419daec1fa7ce100516bd72d0 | |
| parent | 3de24ae2ed6c58fc96a7e50832afe42fe7af34fb (diff) | |
| download | spark-48ee16d8012602c75d50aa2a85e26b7de3c48944.tar.gz spark-48ee16d8012602c75d50aa2a85e26b7de3c48944.tar.bz2 spark-48ee16d8012602c75d50aa2a85e26b7de3c48944.zip | |
[SPARK-14055] writeLocksByTask need to be update when removeBlock
## What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/SPARK-14055
## How was this patch tested?
manual tests by running LiveJournalPageRank on a large dataset ( the dataset must larger enough to incure RDD partition eviction).
Author: Ernest <earneyzxl@gmail.com>
Closes #11875 from Earne/issue-14055.
| -rw-r--r-- | core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala b/core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala index 94d11c5be5..ca53534b61 100644 --- a/core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala +++ b/core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala @@ -421,6 +421,7 @@ private[storage] class BlockInfoManager extends Logging { infos.remove(blockId) blockInfo.readerCount = 0 blockInfo.writerTask = BlockInfo.NO_WRITER + writeLocksByTask.removeBinding(currentTaskAttemptId, blockId) } case None => throw new IllegalArgumentException( |