diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2015-08-04 14:42:11 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-08-04 14:42:11 -0700 |
| commit | ab8ee1a3b93286a62949569615086ef5030e9fae (patch) | |
| tree | 88aa364451320f2a303bbeeb4857bcba57896c84 /sql/catalyst | |
| parent | f4b1ac08a1327e6d0ddc317cdf3997a0f68dec72 (diff) | |
| download | spark-ab8ee1a3b93286a62949569615086ef5030e9fae.tar.gz spark-ab8ee1a3b93286a62949569615086ef5030e9fae.tar.bz2 spark-ab8ee1a3b93286a62949569615086ef5030e9fae.zip | |
[SPARK-9452] [SQL] Support records larger than page size in UnsafeExternalSorter
This patch extends UnsafeExternalSorter to support records larger than the page size. The basic strategy is the same as in #7762: store large records in their own overflow pages.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #7891 from JoshRosen/large-records-in-sql-sorter and squashes the following commits:
967580b [Josh Rosen] Merge remote-tracking branch 'origin/master' into large-records-in-sql-sorter
948c344 [Josh Rosen] Add large records tests for KV sorter.
3c17288 [Josh Rosen] Combine memory and disk cleanup into general cleanupResources() method
380f217 [Josh Rosen] Merge remote-tracking branch 'origin/master' into large-records-in-sql-sorter
27eafa0 [Josh Rosen] Fix page size in PackedRecordPointerSuite
a49baef [Josh Rosen] Address initial round of review comments
3edb931 [Josh Rosen] Remove accidentally-committed debug statements.
2b164e2 [Josh Rosen] Support large records in UnsafeExternalSorter.
Diffstat (limited to 'sql/catalyst')
| -rw-r--r-- | sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java b/sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java index 193906d247..a5ae2b9736 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java @@ -114,7 +114,7 @@ final class UnsafeExternalRowSorter { } private void cleanupResources() { - sorter.freeMemory(); + sorter.cleanupResources(); } @VisibleForTesting |