diff options
author | jerryshao <sshao@hortonworks.com> | 2016-01-13 16:34:23 -0800 |
---|---|---|
committer | Josh Rosen <joshrosen@databricks.com> | 2016-01-13 16:34:23 -0800 |
commit | cd81fc9e8652c07b84f0887a24d67381b4e605fa (patch) | |
tree | 83f75886b61d38a0650101f8dfddcd418ad0b018 /R/WINDOWS.md | |
parent | eabc7b8ee7e809bab05361ed154f87bff467bd88 (diff) | |
download | spark-cd81fc9e8652c07b84f0887a24d67381b4e605fa.tar.gz spark-cd81fc9e8652c07b84f0887a24d67381b4e605fa.tar.bz2 spark-cd81fc9e8652c07b84f0887a24d67381b4e605fa.zip |
[SPARK-12400][SHUFFLE] Avoid generating temp shuffle files for empty partitions
This problem lies in `BypassMergeSortShuffleWriter`, empty partition will also generate a temp shuffle file with several bytes. So here change to only create file when partition is not empty.
This problem only lies in here, no such issue in `HashShuffleWriter`.
Please help to review, thanks a lot.
Author: jerryshao <sshao@hortonworks.com>
Closes #10376 from jerryshao/SPARK-12400.
Diffstat (limited to 'R/WINDOWS.md')
0 files changed, 0 insertions, 0 deletions