diff options
author | Davies Liu <davies@databricks.com> | 2015-11-05 19:02:18 -0800 |
---|---|---|
committer | Josh Rosen <joshrosen@databricks.com> | 2015-11-05 19:02:18 -0800 |
commit | eec74ba8bde7f9446cc38e687bda103e85669d35 (patch) | |
tree | 5e656d6333afde0255e96d930b245df28994bf9b /sql | |
parent | 3cc2c053b5d68c747a30bd58cf388b87b1922f13 (diff) | |
download | spark-eec74ba8bde7f9446cc38e687bda103e85669d35.tar.gz spark-eec74ba8bde7f9446cc38e687bda103e85669d35.tar.bz2 spark-eec74ba8bde7f9446cc38e687bda103e85669d35.zip |
[SPARK-7542][SQL] Support off-heap index/sort buffer
This brings the support of off-heap memory for array inside BytesToBytesMap and InMemorySorter, then we could allocate all the memory from off-heap for execution.
Closes #8068
Author: Davies Liu <davies@databricks.com>
Closes #9477 from davies/unsafe_timsort.
Diffstat (limited to 'sql')
-rw-r--r-- | sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java b/sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java index e2898ef2e2..8c9b9c85e3 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java @@ -85,8 +85,9 @@ public final class UnsafeKVExternalSorter { } else { // During spilling, the array in map will not be used, so we can borrow that and use it // as the underline array for in-memory sorter (it's always large enough). + // Since we will not grow the array, it's fine to pass `null` as consumer. final UnsafeInMemorySorter inMemSorter = new UnsafeInMemorySorter( - taskMemoryManager, recordComparator, prefixComparator, map.getArray()); + null, taskMemoryManager, recordComparator, prefixComparator, map.getArray()); // We cannot use the destructive iterator here because we are reusing the existing memory // pages in BytesToBytesMap to hold records during sorting. |