From 4f369176b750c980682a6be468cefa8627769c72 Mon Sep 17 00:00:00 2001 From: Lianhui Wang Date: Thu, 21 Apr 2016 10:02:23 -0700 Subject: [SPARK-4452] [CORE] Shuffle data structures can starve others on the same thread for memory ## What changes were proposed in this pull request? In #9241 It implemented a mechanism to call spill() on those SQL operators that support spilling if there is not enough memory for execution. But ExternalSorter and AppendOnlyMap in Spark core are not worked. So this PR make them benefit from #9241. Now when there is not enough memory for execution, it can get memory by spilling ExternalSorter and AppendOnlyMap in Spark core. ## How was this patch tested? add two unit tests for it. Author: Lianhui Wang Closes #10024 from lianhuiwang/SPARK-4452-2. --- project/MimaExcludes.scala | 3 +++ 1 file changed, 3 insertions(+) (limited to 'project/MimaExcludes.scala') diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala index 65b0a97e4d..3c9f1532f9 100644 --- a/project/MimaExcludes.scala +++ b/project/MimaExcludes.scala @@ -660,6 +660,9 @@ object MimaExcludes { // SPARK-14704: Create accumulators in TaskMetrics ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.InputMetrics.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.OutputMetrics.this") + ) ++ Seq( + // [SPARK-4452][Core]Shuffle data structures can starve others on the same thread for memory + ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.util.collection.Spillable") ) case v if v.startsWith("1.6") => Seq( -- cgit v1.2.3