diff options
| author | Andrew Or <andrew@databricks.com> | 2015-09-18 23:58:25 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-09-18 23:58:25 -0700 |
| commit | 7ff8d68cc19299e16dedfd819b9e96480fa6cf44 (patch) | |
| tree | bf4851837bc99d35cdc6ac2c00ad46511835affb /docs/sql-programming-guide.md | |
| parent | 22be2ae147a111e88896f6fb42ed46bbf108a99b (diff) | |
| download | spark-7ff8d68cc19299e16dedfd819b9e96480fa6cf44.tar.gz spark-7ff8d68cc19299e16dedfd819b9e96480fa6cf44.tar.bz2 spark-7ff8d68cc19299e16dedfd819b9e96480fa6cf44.zip | |
[SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array
When `TungstenAggregation` hits memory pressure, it switches from hash-based to sort-based aggregation in-place. However, in the process we try to allocate the pointer array for writing to the new `UnsafeExternalSorter` *before* actually freeing the memory from the hash map. This lead to the following exception:
```
java.io.IOException: Could not acquire 65536 bytes of memory
at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.initializeForWriting(UnsafeExternalSorter.java:169)
at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill(UnsafeExternalSorter.java:220)
at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:126)
at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:257)
at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.switchToSortBasedAggregation(TungstenAggregationIterator.scala:435)
```
Author: Andrew Or <andrew@databricks.com>
Closes #8827 from andrewor14/allocate-pointer-array.
Diffstat (limited to 'docs/sql-programming-guide.md')
0 files changed, 0 insertions, 0 deletions