diff options
| author | Davies Liu <davies@databricks.com> | 2016-06-03 16:45:09 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-06-03 16:45:09 -0700 |
| commit | 3074f575a3c84108fddab3f5f56eb1929a4b2cff (patch) | |
| tree | 381fdb85dd2b52bc79b0e18c27eded1bd66d622d /yarn/src | |
| parent | 67cc89ff028324ba4a7a7d9c19a268b9afea0031 (diff) | |
| download | spark-3074f575a3c84108fddab3f5f56eb1929a4b2cff.tar.gz spark-3074f575a3c84108fddab3f5f56eb1929a4b2cff.tar.bz2 spark-3074f575a3c84108fddab3f5f56eb1929a4b2cff.zip | |
[SPARK-15391] [SQL] manage the temporary memory of timsort
## What changes were proposed in this pull request?
Currently, the memory for temporary buffer used by TimSort is always allocated as on-heap without bookkeeping, it could cause OOM both in on-heap and off-heap mode.
This PR will try to manage that by preallocate it together with the pointer array, same with RadixSort. It both works for on-heap and off-heap mode.
This PR also change the loadFactor of BytesToBytesMap to 0.5 (it was 0.70), it enables use to radix sort also makes sure that we have enough memory for timsort.
## How was this patch tested?
Existing tests.
Author: Davies Liu <davies@databricks.com>
Closes #13318 from davies/fix_timsort.
Diffstat (limited to 'yarn/src')
0 files changed, 0 insertions, 0 deletions