diff options
| author | Liang-Chi Hsieh <viirya@appier.com> | 2015-08-06 14:33:29 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-08-06 14:33:29 -0700 |
| commit | 21fdfd7d6f89adbd37066c169e6ba9ccd337683e (patch) | |
| tree | 6d32717cad457f0037ee332dc51a807834d59f76 /streaming | |
| parent | abfedb9cd70af60c8290bd2f5a5cec1047845ba0 (diff) | |
| download | spark-21fdfd7d6f89adbd37066c169e6ba9ccd337683e.tar.gz spark-21fdfd7d6f89adbd37066c169e6ba9ccd337683e.tar.bz2 spark-21fdfd7d6f89adbd37066c169e6ba9ccd337683e.zip | |
[SPARK-9548][SQL] Add a destructive iterator for BytesToBytesMap
This pull request adds a destructive iterator to BytesToBytesMap. When used, the iterator frees pages as it traverses them. This is part of the effort to avoid starving when we have more than one operators that can exhaust memory.
This is based on #7924, but fixes a bug there (Don't use destructive iterator in UnsafeKVExternalSorter).
Closes #7924.
Author: Liang-Chi Hsieh <viirya@appier.com>
Author: Reynold Xin <rxin@databricks.com>
Closes #8003 from rxin/map-destructive-iterator and squashes the following commits:
6b618c3 [Reynold Xin] Don't use destructive iterator in UnsafeKVExternalSorter.
a7bd8ec [Reynold Xin] Merge remote-tracking branch 'viirya/destructive_iter' into map-destructive-iterator
7652083 [Liang-Chi Hsieh] For comments: add destructiveIterator(), modify unit test, remove code block.
4a3e9de [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into destructive_iter
581e9e3 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into destructive_iter
f0ff783 [Liang-Chi Hsieh] No need to free last page.
9e9d2a3 [Liang-Chi Hsieh] Add a destructive iterator for BytesToBytesMap.
Diffstat (limited to 'streaming')
0 files changed, 0 insertions, 0 deletions