From ad9e3d50f71b096872806a86d89c03a208b1cf8b Mon Sep 17 00:00:00 2001 From: jeanlyn Date: Mon, 28 Mar 2016 16:56:25 -0700 Subject: [SPARK-13845][CORE] Using onBlockUpdated to replace onTaskEnd avioding driver OOM ## What changes were proposed in this pull request? We have a streaming job using `FlumePollInputStream` always driver OOM after few days, here is some driver heap dump before OOM ``` num #instances #bytes class name ---------------------------------------------- 1: 13845916 553836640 org.apache.spark.storage.BlockStatus 2: 14020324 336487776 org.apache.spark.storage.StreamBlockId 3: 13883881 333213144 scala.collection.mutable.DefaultEntry 4: 8907 89043952 [Lscala.collection.mutable.HashEntry; 5: 62360 65107352 [B 6: 163368 24453904 [Ljava.lang.Object; 7: 293651 20342664 [C ... ``` `BlockStatus` and `StreamBlockId` keep on growing, and the driver OOM in the end. After investigated, i found the `executorIdToStorageStatus` in `StorageStatusListener` seems never remove the blocks from `StorageStatus`. In order to fix the issue, i try to use `onBlockUpdated` replace `onTaskEnd ` , so we can update the block informations(add blocks, drop the block from memory to disk and delete the blocks) in time. ## How was this patch tested? Existing unit tests and manual tests Author: jeanlyn Closes #11779 from jeanlyn/fix_driver_oom. --- .../HistoryServerExpectations/executor_list_json_expectation.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json') diff --git a/core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json index 4a88eeee74..efc865919b 100644 --- a/core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json +++ b/core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json @@ -2,8 +2,8 @@ "id" : "", "hostPort" : "localhost:57971", "isActive" : true, - "rddBlocks" : 8, - "memoryUsed" : 28000128, + "rddBlocks" : 0, + "memoryUsed" : 0, "diskUsed" : 0, "totalCores" : 0, "maxTasks" : 0, -- cgit v1.2.3