diff options
| author | Andrew Or <andrew@databricks.com> | 2016-04-07 17:49:39 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2016-04-07 17:49:39 -0700 |
| commit | 3e29e372ff518827bae9dcd26087946fde476843 (patch) | |
| tree | 3abd3e22678fb63e347832e1241bf94fd2a8e6b9 /project | |
| parent | 30e980ad8e6443dddd54f3c2d48b3904499545cf (diff) | |
| download | spark-3e29e372ff518827bae9dcd26087946fde476843.tar.gz spark-3e29e372ff518827bae9dcd26087946fde476843.tar.bz2 spark-3e29e372ff518827bae9dcd26087946fde476843.zip | |
[SPARK-14468] Always enable OutputCommitCoordinator
## What changes were proposed in this pull request?
`OutputCommitCoordinator` was introduced to deal with concurrent task attempts racing to write output, leading to data loss or corruption. For more detail, read the [JIRA description](https://issues.apache.org/jira/browse/SPARK-14468).
Before: `OutputCommitCoordinator` is enabled only if speculation is enabled.
After: `OutputCommitCoordinator` is always enabled.
Users may still disable this through `spark.hadoop.outputCommitCoordination.enabled`, but they really shouldn't...
## How was this patch tested?
`OutputCommitCoordinator*Suite`
Author: Andrew Or <andrew@databricks.com>
Closes #12244 from andrewor14/always-occ.
Diffstat (limited to 'project')
0 files changed, 0 insertions, 0 deletions