diff options
| author | Lianhui Wang <lianhuiwang09@gmail.com> | 2016-04-21 10:02:23 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-04-21 10:02:23 -0700 |
| commit | 4f369176b750c980682a6be468cefa8627769c72 (patch) | |
| tree | 6afcbb5af6f541e660cf756dfaff192f4285d45d /project | |
| parent | 649335d6c1dd0884bf6f6618cd56fb480fe40886 (diff) | |
| download | spark-4f369176b750c980682a6be468cefa8627769c72.tar.gz spark-4f369176b750c980682a6be468cefa8627769c72.tar.bz2 spark-4f369176b750c980682a6be468cefa8627769c72.zip | |
[SPARK-4452] [CORE] Shuffle data structures can starve others on the same thread for memory
## What changes were proposed in this pull request?
In #9241 It implemented a mechanism to call spill() on those SQL operators that support spilling if there is not enough memory for execution.
But ExternalSorter and AppendOnlyMap in Spark core are not worked. So this PR make them benefit from #9241. Now when there is not enough memory for execution, it can get memory by spilling ExternalSorter and AppendOnlyMap in Spark core.
## How was this patch tested?
add two unit tests for it.
Author: Lianhui Wang <lianhuiwang09@gmail.com>
Closes #10024 from lianhuiwang/SPARK-4452-2.
Diffstat (limited to 'project')
| -rw-r--r-- | project/MimaExcludes.scala | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala index 65b0a97e4d..3c9f1532f9 100644 --- a/project/MimaExcludes.scala +++ b/project/MimaExcludes.scala @@ -660,6 +660,9 @@ object MimaExcludes { // SPARK-14704: Create accumulators in TaskMetrics ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.InputMetrics.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.OutputMetrics.this") + ) ++ Seq( + // [SPARK-4452][Core]Shuffle data structures can starve others on the same thread for memory + ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.util.collection.Spillable") ) case v if v.startsWith("1.6") => Seq( |