diff options
| author | jerryshao <saisai.shao@intel.com> | 2014-10-20 10:20:21 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2014-10-20 10:20:21 -0700 |
| commit | c7aeecd08fd329085760fa89025ec0d9c04f5e3f (patch) | |
| tree | 82d9bc132017bf3577e477488e5e39ff7b74e048 /repl | |
| parent | d1966f3a8bafdcef87d10ef9db5976cf89faee4b (diff) | |
| download | spark-c7aeecd08fd329085760fa89025ec0d9c04f5e3f.tar.gz spark-c7aeecd08fd329085760fa89025ec0d9c04f5e3f.tar.bz2 spark-c7aeecd08fd329085760fa89025ec0d9c04f5e3f.zip | |
[SPARK-3948][Shuffle]Fix stream corruption bug in sort-based shuffle
Kernel 2.6.32 bug will lead to unexpected behavior of transferTo in copyStream, and this will corrupt the shuffle output file in sort-based shuffle, which will somehow introduce PARSING_ERROR(2), deserialization error or offset out of range. Here fix this by adding append flag, also add some position checking code. Details can be seen in [SPARK-3948](https://issues.apache.org/jira/browse/SPARK-3948).
Author: jerryshao <saisai.shao@intel.com>
Closes #2824 from jerryshao/SPARK-3948 and squashes the following commits:
be0533a [jerryshao] Address the comments
a82b184 [jerryshao] add configuration to control the NIO way of copying stream
e17ada2 [jerryshao] Fix kernel 2.6.32 bug led unexpected behavior of transferTo
Diffstat (limited to 'repl')
0 files changed, 0 insertions, 0 deletions