diff options
| author | Eric Liang <ekl@databricks.com> | 2016-09-08 16:47:18 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2016-09-08 16:47:18 -0700 |
| commit | 722afbb2b33037a30d385a15725f2db5365bd375 (patch) | |
| tree | 4295f62212e010653b7d5bd82f62149ca5f7a43b /pom.xml | |
| parent | 78d5d4dd5ce5a537ed04cd1bf242c9e9ea2c391a (diff) | |
| download | spark-722afbb2b33037a30d385a15725f2db5365bd375.tar.gz spark-722afbb2b33037a30d385a15725f2db5365bd375.tar.bz2 spark-722afbb2b33037a30d385a15725f2db5365bd375.zip | |
[SPARK-17405] RowBasedKeyValueBatch should use default page size to prevent OOMs
## What changes were proposed in this pull request?
Before this change, we would always allocate 64MB per aggregation task for the first-level hash map storage, even when running in low-memory situations such as local mode. This changes it to use the memory manager default page size, which is automatically reduced from 64MB in these situations.
cc ooq JoshRosen
## How was this patch tested?
Tested manually with `bin/spark-shell --master=local[32]` and verifying that `(1 to math.pow(10, 3).toInt).toDF("n").withColumn("m", 'n % 2).groupBy('m).agg(sum('n)).show` does not crash.
Author: Eric Liang <ekl@databricks.com>
Closes #15016 from ericl/sc-4483.
Diffstat (limited to 'pom.xml')
0 files changed, 0 insertions, 0 deletions