diff options
| author | Eric Liang <ekl@databricks.com> | 2016-12-02 21:59:02 +0800 |
|---|---|---|
| committer | Wenchen Fan <wenchen@databricks.com> | 2016-12-02 21:59:02 +0800 |
| commit | 7935c8470c5c162ef7213e394fe8588e5dd42ca2 (patch) | |
| tree | 0b35862d5113a400a2464c22282f6040b6bcc0d5 /docs/mllib-ensembles.md | |
| parent | 55d528f2ba0ba689dbb881616d9436dc7958e943 (diff) | |
| download | spark-7935c8470c5c162ef7213e394fe8588e5dd42ca2.tar.gz spark-7935c8470c5c162ef7213e394fe8588e5dd42ca2.tar.bz2 spark-7935c8470c5c162ef7213e394fe8588e5dd42ca2.zip | |
[SPARK-18659][SQL] Incorrect behaviors in overwrite table for datasource tables
## What changes were proposed in this pull request?
Two bugs are addressed here
1. INSERT OVERWRITE TABLE sometime crashed when catalog partition management was enabled. This was because when dropping partitions after an overwrite operation, the Hive client will attempt to delete the partition files. If the entire partition directory was dropped, this would fail. The PR fixes this by adding a flag to control whether the Hive client should attempt to delete files.
2. The static partition spec for OVERWRITE TABLE was not correctly resolved to the case-sensitive original partition names. This resulted in the entire table being overwritten if you did not correctly capitalize your partition names.
cc yhuai cloud-fan
## How was this patch tested?
Unit tests. Surprisingly, the existing overwrite table tests did not catch these edge cases.
Author: Eric Liang <ekl@databricks.com>
Closes #16088 from ericl/spark-18659.
Diffstat (limited to 'docs/mllib-ensembles.md')
0 files changed, 0 insertions, 0 deletions