diff options
| author | Matei Zaharia <matei@databricks.com> | 2014-08-04 23:41:03 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-08-04 23:41:03 -0700 |
| commit | 4fde28c2063f673ec7f51d514ba62a73321960a1 (patch) | |
| tree | 1125522786941e5a71ea2526df435bc9f6706405 /docs/spark-standalone.md | |
| parent | 066765d60d21b6b9943862b788e4a4bd07396e6c (diff) | |
| download | spark-4fde28c2063f673ec7f51d514ba62a73321960a1.tar.gz spark-4fde28c2063f673ec7f51d514ba62a73321960a1.tar.bz2 spark-4fde28c2063f673ec7f51d514ba62a73321960a1.zip | |
SPARK-2711. Create a ShuffleMemoryManager to track memory for all spilling collections
This tracks memory properly if there are multiple spilling collections in the same task (which was a problem before), and also implements an algorithm that lets each thread grow up to 1 / 2N of the memory pool (where N is the number of threads) before spilling, which avoids an inefficiency with small spills we had before (some threads would spill many times at 0-1 MB because the pool was allocated elsewhere).
Author: Matei Zaharia <matei@databricks.com>
Closes #1707 from mateiz/spark-2711 and squashes the following commits:
debf75b [Matei Zaharia] Review comments
24f28f3 [Matei Zaharia] Small rename
c8f3a8b [Matei Zaharia] Update ShuffleMemoryManager to be able to partially grant requests
315e3a5 [Matei Zaharia] Some review comments
b810120 [Matei Zaharia] Create central manager to track memory for all spilling collections
Diffstat (limited to 'docs/spark-standalone.md')
0 files changed, 0 insertions, 0 deletions