diff options
| author | Eric Liang <ekl@databricks.com> | 2016-11-02 14:15:10 +0800 |
|---|---|---|
| committer | Wenchen Fan <wenchen@databricks.com> | 2016-11-02 14:15:10 +0800 |
| commit | abefe2ec428dc24a4112c623fb6fbe4b2ca60a2b (patch) | |
| tree | c37c599640ae00418499d9467a025a47daebe43d /core/src | |
| parent | 620da3b4828b3580c7ed7339b2a07938e6be1bb1 (diff) | |
| download | spark-abefe2ec428dc24a4112c623fb6fbe4b2ca60a2b.tar.gz spark-abefe2ec428dc24a4112c623fb6fbe4b2ca60a2b.tar.bz2 spark-abefe2ec428dc24a4112c623fb6fbe4b2ca60a2b.zip | |
[SPARK-18183][SPARK-18184] Fix INSERT [INTO|OVERWRITE] TABLE ... PARTITION for Datasource tables
## What changes were proposed in this pull request?
There are a couple issues with the current 2.1 behavior when inserting into Datasource tables with partitions managed by Hive.
(1) OVERWRITE TABLE ... PARTITION will actually overwrite the entire table instead of just the specified partition.
(2) INSERT|OVERWRITE does not work with partitions that have custom locations.
This PR fixes both of these issues for Datasource tables managed by Hive. The behavior for legacy tables or when `manageFilesourcePartitions = false` is unchanged.
There is one other issue in that INSERT OVERWRITE with dynamic partitions will overwrite the entire table instead of just the updated partitions, but this behavior is pretty complicated to implement for Datasource tables. We should address that in a future release.
## How was this patch tested?
Unit tests.
Author: Eric Liang <ekl@databricks.com>
Closes #15705 from ericl/sc-4942.
Diffstat (limited to 'core/src')
0 files changed, 0 insertions, 0 deletions