diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2015-11-04 17:19:52 -0800 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2015-11-04 17:19:52 -0800 |
| commit | d0b56339625727744e2c30fc2167bc6a457d37f7 (patch) | |
| tree | fc3b347ca67cdaaf826e362b45360ee78da5d6fa /sql/hive | |
| parent | a752ddad7fe1d0f01b51f7551ec017ff87e1eea5 (diff) | |
| download | spark-d0b56339625727744e2c30fc2167bc6a457d37f7.tar.gz spark-d0b56339625727744e2c30fc2167bc6a457d37f7.tar.bz2 spark-d0b56339625727744e2c30fc2167bc6a457d37f7.zip | |
[SPARK-11307] Reduce memory consumption of OutputCommitCoordinator
OutputCommitCoordinator uses a map in a place where an array would suffice, increasing its memory consumption for result stages with millions of tasks.
This patch replaces that map with an array. The only tricky part of this is reasoning about the range of possible array indexes in order to make sure that we never index out of bounds.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #9274 from JoshRosen/SPARK-11307.
Diffstat (limited to 'sql/hive')
0 files changed, 0 insertions, 0 deletions