diff options
author | Reynold Xin <rxin@databricks.com> | 2015-08-02 12:32:14 -0700 |
---|---|---|
committer | Josh Rosen <joshrosen@databricks.com> | 2015-08-02 12:32:14 -0700 |
commit | 2e981b7bfa9dec93fdcf25f3e7220cd6aaba744f (patch) | |
tree | f7458ae297d36bba1acf21fd08169defef6c2ef8 /sql/core/pom.xml | |
parent | 66924ffa6bdb8e0df1b90b789cb7ad443377e729 (diff) | |
download | spark-2e981b7bfa9dec93fdcf25f3e7220cd6aaba744f.tar.gz spark-2e981b7bfa9dec93fdcf25f3e7220cd6aaba744f.tar.bz2 spark-2e981b7bfa9dec93fdcf25f3e7220cd6aaba744f.zip |
[SPARK-9531] [SQL] UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter
This pull request adds a destructAndCreateExternalSorter method to UnsafeFixedWidthAggregationMap. The new method does the following:
1. Creates a new external sorter UnsafeKVExternalSorter
2. Adds all the data into an in-memory sorter, sorts them
3. Spills the sorted in-memory data to disk
This method can be used to fallback to sort-based aggregation when under memory pressure.
The pull request also includes accounting fixes from JoshRosen.
TODOs (that can be done in follow-up PRs)
- [x] Address Josh's feedbacks from #7849
- [x] More documentation and test cases
- [x] Make sure we are doing memory accounting correctly with test cases (e.g. did we release the memory in BytesToBytesMap twice?)
- [ ] Look harder at possible memory leaks and exception handling
- [ ] Randomized tester for the KV sorter as well as the aggregation map
Author: Reynold Xin <rxin@databricks.com>
Author: Josh Rosen <joshrosen@databricks.com>
Closes #7860 from rxin/kvsorter and squashes the following commits:
986a58c [Reynold Xin] Bug fix.
599317c [Reynold Xin] Style fix and slightly more compact code.
fe7bd4e [Reynold Xin] Bug fixes.
fd71bef [Reynold Xin] Merge remote-tracking branch 'josh/large-records-in-sql-sorter' into kvsorter-with-josh-fix
3efae38 [Reynold Xin] More fixes and documentation.
45f1b09 [Josh Rosen] Ensure that spill files are cleaned up
f6a9bd3 [Reynold Xin] Josh feedback.
9be8139 [Reynold Xin] Remove testSpillFrequency.
7cbe759 [Reynold Xin] [SPARK-9531][SQL] UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter.
ae4a8af [Josh Rosen] Detect leaked unsafe memory in UnsafeExternalSorterSuite.
52f9b06 [Josh Rosen] Detect ShuffleMemoryManager leaks in UnsafeExternalSorter.
Diffstat (limited to 'sql/core/pom.xml')
-rw-r--r-- | sql/core/pom.xml | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/sql/core/pom.xml b/sql/core/pom.xml index be0966641b..349007789f 100644 --- a/sql/core/pom.xml +++ b/sql/core/pom.xml @@ -106,6 +106,11 @@ <artifactId>parquet-avro</artifactId> <scope>test</scope> </dependency> + <dependency> + <groupId>org.mockito</groupId> + <artifactId>mockito-core</artifactId> + <scope>test</scope> + </dependency> </dependencies> <build> <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory> |