diff options
author | huangzhaowei <carlmartinmax@gmail.com> | 2016-08-11 14:56:03 -0700 |
---|---|---|
committer | Marcelo Vanzin <vanzin@cloudera.com> | 2016-08-11 14:56:03 -0700 |
commit | 4ec5c360ce2045a9bdecb3c5277ba519bf0f44ae (patch) | |
tree | c78bfd6d101971467c2d6e5f1a99420a638fda06 /core | |
parent | 1c9a386c6b6812a3931f3fb0004249894a01f657 (diff) | |
download | spark-4ec5c360ce2045a9bdecb3c5277ba519bf0f44ae.tar.gz spark-4ec5c360ce2045a9bdecb3c5277ba519bf0f44ae.tar.bz2 spark-4ec5c360ce2045a9bdecb3c5277ba519bf0f44ae.zip |
[SPARK-16868][WEB UI] Fix executor be both dead and alive on executor ui.
## What changes were proposed in this pull request?
In a heavy pressure of the spark application, since the executor will register it to driver block manager twice(because of heart beats), the executor will show as picture show:
![image](https://cloud.githubusercontent.com/assets/7404824/17467245/c1359094-5d4e-11e6-843a-f6d6347e1bf6.png)
## How was this patch tested?
NA
Details in: [SPARK-16868](https://issues.apache.org/jira/browse/SPARK-16868)
Author: huangzhaowei <carlmartinmax@gmail.com>
Closes #14530 from SaintBacchus/SPARK-16868.
Diffstat (limited to 'core')
-rw-r--r-- | core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala index 3008520f61..798658a15b 100644 --- a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala +++ b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala @@ -77,6 +77,10 @@ class StorageStatusListener(conf: SparkConf) extends SparkListener { val maxMem = blockManagerAdded.maxMem val storageStatus = new StorageStatus(blockManagerId, maxMem) executorIdToStorageStatus(executorId) = storageStatus + + // Try to remove the dead storage status if same executor register the block manager twice. + deadExecutorStorageStatus.zipWithIndex.find(_._1.blockManagerId.executorId == executorId) + .foreach(toRemoveExecutor => deadExecutorStorageStatus.remove(toRemoveExecutor._2)) } } |