diff options
| author | jeanlyn <jeanlyn92@gmail.com> | 2016-03-31 12:04:42 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2016-03-31 12:04:42 -0700 |
| commit | 8a333d2da859fd593bda183413630bc3757529c9 (patch) | |
| tree | 646fac41a0dd46838b89966dbff3cc9ccbbf4b9f /core/src/main/scala/org/apache | |
| parent | 446c45bd87035e20653394fcaf9dc8caa4299038 (diff) | |
| download | spark-8a333d2da859fd593bda183413630bc3757529c9.tar.gz spark-8a333d2da859fd593bda183413630bc3757529c9.tar.bz2 spark-8a333d2da859fd593bda183413630bc3757529c9.zip | |
[SPARK-14243][CORE] update task metrics when removing blocks
## What changes were proposed in this pull request?
This PR try to use `incUpdatedBlockStatuses ` to update the `updatedBlockStatuses ` when removing blocks, making sure `BlockManager` correctly updates `updatedBlockStatuses`
## How was this patch tested?
test("updated block statuses") in BlockManagerSuite.scala
Author: jeanlyn <jeanlyn92@gmail.com>
Closes #12091 from jeanlyn/updateBlock.
Diffstat (limited to 'core/src/main/scala/org/apache')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/storage/BlockManager.scala | 7 |
1 files changed, 5 insertions, 2 deletions
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala index 0c7763f236..3014cafc28 100644 --- a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala +++ b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala @@ -1264,9 +1264,12 @@ private[spark] class BlockManager( "the disk, memory, or external block store") } blockInfoManager.removeBlock(blockId) + val removeBlockStatus = getCurrentBlockStatus(blockId, info) if (tellMaster && info.tellMaster) { - val status = getCurrentBlockStatus(blockId, info) - reportBlockStatus(blockId, info, status) + reportBlockStatus(blockId, info, removeBlockStatus) + } + Option(TaskContext.get()).foreach { c => + c.taskMetrics().incUpdatedBlockStatuses(Seq((blockId, removeBlockStatus))) } } } |