diff options
| author | gatorsmile <gatorsmile@gmail.com> | 2016-05-12 11:14:40 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2016-05-12 11:14:40 -0700 |
| commit | be617f3d0695982f982006fdd79afe3e3730b4c4 (patch) | |
| tree | d15e5bc1f502c4cf61b866ccc12ec45f7de64967 /streaming | |
| parent | 470de743ecf3617babd86f50ab203e85aa975d69 (diff) | |
| download | spark-be617f3d0695982f982006fdd79afe3e3730b4c4.tar.gz spark-be617f3d0695982f982006fdd79afe3e3730b4c4.tar.bz2 spark-be617f3d0695982f982006fdd79afe3e3730b4c4.zip | |
[SPARK-14684][SPARK-15277][SQL] Partition Spec Validation in SessionCatalog and Checking Partition Spec Existence Before Dropping
#### What changes were proposed in this pull request?
~~Currently, multiple partitions are allowed to drop by using a single DDL command: Alter Table Drop Partition. However, the internal implementation could break atomicity. That means, we could just drop a subset of qualified partitions, if hitting an exception when dropping one of qualified partitions~~
~~This PR contains the following behavior changes:~~
~~- disallow dropping multiple partitions by a single command ~~
~~- allow users to input predicates in partition specification and issue a nicer error message if the predicate's comparison operator is not `=`.~~
~~- verify the partition spec in SessionCatalog. This can ensure each partition spec in `Drop Partition` does not correspond to multiple partitions.~~
This PR has two major parts:
- Verify the partition spec in SessionCatalog for fixing the following issue:
```scala
sql(s"ALTER TABLE $externalTab DROP PARTITION (ds='2008-04-09', unknownCol='12')")
```
Above example uses an invalid partition spec. Without this PR, we will drop all the partitions. The reason is Hive megastores getPartitions API returns all the partitions if we provide an invalid spec.
- Re-implemented the `dropPartitions` in `HiveClientImpl`. Now, we always check if all the user-specified partition specs exist before attempting to drop the partitions. Previously, we start drop the partition before completing checking the existence of all the partition specs. If any failure happened after we start to drop the partitions, we will log an error message to indicate which partitions have been dropped and which partitions have not been dropped.
#### How was this patch tested?
Modified the existing test cases and added new test cases.
Author: gatorsmile <gatorsmile@gmail.com>
Author: xiaoli <lixiao1983@gmail.com>
Author: Xiao Li <xiaoli@Xiaos-MacBook-Pro.local>
Closes #12801 from gatorsmile/banDropMultiPart.
Diffstat (limited to 'streaming')
0 files changed, 0 insertions, 0 deletions