diff options
| author | Davies Liu <davies@databricks.com> | 2015-08-31 15:55:22 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-08-31 15:55:29 -0700 |
| commit | 1c752b8b5c7090936b5c2ca94e8fb47c4f570d69 (patch) | |
| tree | 042116e9d3cb284a30eb14d72c583bb27f3ba4b4 /python/pyspark/sql.py | |
| parent | 33ce274cdf7538b5816f1a400b2fad394ec2a056 (diff) | |
| download | spark-1c752b8b5c7090936b5c2ca94e8fb47c4f570d69.tar.gz spark-1c752b8b5c7090936b5c2ca94e8fb47c4f570d69.tar.bz2 spark-1c752b8b5c7090936b5c2ca94e8fb47c4f570d69.zip | |
[SPARK-10341] [SQL] fix memory starving in unsafe SMJ
In SMJ, the first ExternalSorter could consume all the memory before spilling, then the second can not even acquire the first page.
Before we have a better memory allocator, SMJ should call prepare() before call any compute() of it's children.
cc rxin JoshRosen
Author: Davies Liu <davies@databricks.com>
Closes #8511 from davies/smj_memory.
(cherry picked from commit 540bdee93103a73736d282b95db6a8cda8f6a2b1)
Signed-off-by: Reynold Xin <rxin@databricks.com>
Diffstat (limited to 'python/pyspark/sql.py')
0 files changed, 0 insertions, 0 deletions