diff options
| author | Yin Huai <yhuai@databricks.com> | 2015-06-24 09:50:03 -0700 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2015-06-24 09:50:03 -0700 |
| commit | bba6699d0e9093bc041a9a33dd31992790f32174 (patch) | |
| tree | 0531f84b232f250709e640c2d0d0f6d6a97a7bc3 /ec2 | |
| parent | 9d36ec24312f0a9865b4392f89e9611a5b80916d (diff) | |
| download | spark-bba6699d0e9093bc041a9a33dd31992790f32174.tar.gz spark-bba6699d0e9093bc041a9a33dd31992790f32174.tar.bz2 spark-bba6699d0e9093bc041a9a33dd31992790f32174.zip | |
[SPARK-8578] [SQL] Should ignore user defined output committer when appending data
https://issues.apache.org/jira/browse/SPARK-8578
It is not very safe to use a custom output committer when append data to an existing dir. This changes adds the logic to check if we are appending data, and if so, we use the output committer associated with the file output format.
Author: Yin Huai <yhuai@databricks.com>
Closes #6964 from yhuai/SPARK-8578 and squashes the following commits:
43544c4 [Yin Huai] Do not use a custom output commiter when appendiing data.
Diffstat (limited to 'ec2')
0 files changed, 0 insertions, 0 deletions