diff options
| author | Cheng Lian <lian@databricks.com> | 2014-11-24 12:43:45 -0800 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2014-11-24 12:43:58 -0800 |
| commit | ee1bc892a32bb969b051b3bc3eaaf9a54af1c7a3 (patch) | |
| tree | 0d5034fadfdcf6fdd480f6f597b234e87e7a27c3 /sql/hive-thriftserver | |
| parent | 1a12ca339cf038c44f5d7402d63851f48a055b35 (diff) | |
| download | spark-ee1bc892a32bb969b051b3bc3eaaf9a54af1c7a3.tar.gz spark-ee1bc892a32bb969b051b3bc3eaaf9a54af1c7a3.tar.bz2 spark-ee1bc892a32bb969b051b3bc3eaaf9a54af1c7a3.zip | |
[SPARK-4479][SQL] Avoids unnecessary defensive copies when sort based shuffle is on
This PR is a workaround for SPARK-4479. Two changes are introduced: when merge sort is bypassed in `ExternalSorter`,
1. also bypass RDD elements buffering as buffering is the reason that `MutableRow` backed row objects must be copied, and
2. avoids defensive copies in `Exchange` operator
<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/3422)
<!-- Reviewable:end -->
Author: Cheng Lian <lian@databricks.com>
Closes #3422 from liancheng/avoids-defensive-copies and squashes the following commits:
591f2e9 [Cheng Lian] Passes all shuffle suites
0c3c91e [Cheng Lian] Fixes shuffle write metrics when merge sort is bypassed
ed5df3c [Cheng Lian] Fixes styling changes
f75089b [Cheng Lian] Avoids unnecessary defensive copies when sort based shuffle is on
(cherry picked from commit a6d7b61f92dc7c1f9632cecb232afa8040ab2b4d)
Signed-off-by: Michael Armbrust <michael@databricks.com>
Diffstat (limited to 'sql/hive-thriftserver')
0 files changed, 0 insertions, 0 deletions