diff options
| author | Andrew Or <andrew@databricks.com> | 2015-08-06 19:04:57 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-08-06 19:04:57 -0700 |
| commit | 014a9f9d8c9521180f7a448cc7cc96cc00537d5c (patch) | |
| tree | ffbe979c71cc0811103b75ff35a7131f648adbae /docs/sql-programming-guide.md | |
| parent | b87825310ac87485672868bf6a9ed01d154a3626 (diff) | |
| download | spark-014a9f9d8c9521180f7a448cc7cc96cc00537d5c.tar.gz spark-014a9f9d8c9521180f7a448cc7cc96cc00537d5c.tar.bz2 spark-014a9f9d8c9521180f7a448cc7cc96cc00537d5c.zip | |
[SPARK-9709] [SQL] Avoid starving unsafe operators that use sort
The issue is that a task may run multiple sorts, and the sorts run by the child operator (i.e. parent RDD) may acquire all available memory such that other sorts in the same task do not have enough to proceed. This manifests itself in an `IOException("Unable to acquire X bytes of memory")` thrown by `UnsafeExternalSorter`.
The solution is to reserve a page in each sorter in the chain before computing the child operator's (parent RDD's) partitions. This requires us to use a new special RDD that does some preparation before computing the parent's partitions.
Author: Andrew Or <andrew@databricks.com>
Closes #8011 from andrewor14/unsafe-starve-memory and squashes the following commits:
35b69a4 [Andrew Or] Simplify test
0b07782 [Andrew Or] Minor: update comments
5d5afdf [Andrew Or] Merge branch 'master' of github.com:apache/spark into unsafe-starve-memory
254032e [Andrew Or] Add tests
234acbd [Andrew Or] Reserve a page in sorter when preparing each partition
b889e08 [Andrew Or] MapPartitionsWithPreparationRDD
Diffstat (limited to 'docs/sql-programming-guide.md')
0 files changed, 0 insertions, 0 deletions