diff options
| author | Andrew Or <andrew@databricks.com> | 2015-05-21 14:33:11 -0700 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2015-05-21 14:33:11 -0700 |
| commit | 5287eec5a6948c0c6e0baaebf35f512324c0679a (patch) | |
| tree | 078d0af874331e537667fe5ad8ee5fa3ab6023bb /unsafe | |
| parent | 6b18cdc1b1284b1d48d637d06a1e64829aeb6202 (diff) | |
| download | spark-5287eec5a6948c0c6e0baaebf35f512324c0679a.tar.gz spark-5287eec5a6948c0c6e0baaebf35f512324c0679a.tar.bz2 spark-5287eec5a6948c0c6e0baaebf35f512324c0679a.zip | |
[SPARK-7718] [SQL] Speed up partitioning by avoiding closure cleaning
According to yhuai we spent 6-7 seconds cleaning closures in a partitioning job that takes 12 seconds. Since we provide these closures in Spark we know for sure they are serializable, so we can bypass the cleaning.
Author: Andrew Or <andrew@databricks.com>
Closes #6256 from andrewor14/sql-partition-speed-up and squashes the following commits:
a82b451 [Andrew Or] Fix style
10f7e3e [Andrew Or] Avoid getting call sites and cleaning closures
17e2943 [Andrew Or] Merge branch 'master' of github.com:apache/spark into sql-partition-speed-up
523f042 [Andrew Or] Skip unnecessary Utils.getCallSites too
f7fe143 [Andrew Or] Avoid unnecessary closure cleaning
Diffstat (limited to 'unsafe')
0 files changed, 0 insertions, 0 deletions