diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2015-03-31 16:18:39 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-03-31 16:18:39 -0700 |
| commit | 37326079d818fdb140415a65653767d997613dac (patch) | |
| tree | 6079788d400ba20b3ce05ae2d4c80d24594e85f9 /project/MimaExcludes.scala | |
| parent | 0e00f12d33d28d064c166262b14e012a1aeaa7b0 (diff) | |
| download | spark-37326079d818fdb140415a65653767d997613dac.tar.gz spark-37326079d818fdb140415a65653767d997613dac.tar.bz2 spark-37326079d818fdb140415a65653767d997613dac.zip | |
[SPARK-6614] OutputCommitCoordinator should clear authorized committer only after authorized committer fails, not after any failure
In OutputCommitCoordinator, there is some logic to clear the authorized committer's lock on committing in case that task fails. However, it looks like the current code also clears this lock if other non-authorized tasks fail, which is an obvious bug.
In theory, it's possible that this could allow a new committer to start, run to completion, and commit output before the authorized committer finished, but it's unlikely that this race occurs often in practice due to the complex combination of failure and timing conditions that would be required to expose it.
This patch addresses this issue and adds a regression test.
Thanks to aarondav for spotting this issue.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #5276 from JoshRosen/SPARK-6614 and squashes the following commits:
d532ba7 [Josh Rosen] Check whether failed task was authorized committer
cbb3784 [Josh Rosen] Add regression test for SPARK-6614
Diffstat (limited to 'project/MimaExcludes.scala')
0 files changed, 0 insertions, 0 deletions