aboutsummaryrefslogtreecommitdiff
path: root/sql/hive
diff options
context:
space:
mode:
authorWenchen Fan <wenchen@databricks.com>2016-09-06 14:17:47 +0800
committerWenchen Fan <wenchen@databricks.com>2016-09-06 14:17:47 +0800
commitc0ae6bc6ea38909730fad36e653d3c7ab0a84b44 (patch)
tree25c1e98400ccadb9db221cf02cda423fd0bd7eb4 /sql/hive
parent64e826f91eabb1a22d3d163d71fbb7b6d2185f25 (diff)
downloadspark-c0ae6bc6ea38909730fad36e653d3c7ab0a84b44.tar.gz
spark-c0ae6bc6ea38909730fad36e653d3c7ab0a84b44.tar.bz2
spark-c0ae6bc6ea38909730fad36e653d3c7ab0a84b44.zip
[SPARK-17361][SQL] file-based external table without path should not be created
## What changes were proposed in this pull request? Using the public `Catalog` API, users can create a file-based data source table, without giving the path options. For this case, currently we can create the table successfully, but fail when we read it. Ideally we should fail during creation. This is because when we create data source table, we resolve the data source relation without validating path: `resolveRelation(checkPathExist = false)`. Looking back to why we add this trick(`checkPathExist`), it's because when we call `resolveRelation` for managed table, we add the path to data source options but the path is not created yet. So why we add this not-yet-created path to data source options? This PR fix the problem by adding path to options after we call `resolveRelation`. Then we can remove the `checkPathExist` parameter in `DataSource.resolveRelation` and do some related cleanups. ## How was this patch tested? existing tests and new test in `CatalogSuite` Author: Wenchen Fan <wenchen@databricks.com> Closes #14921 from cloud-fan/check-path.
Diffstat (limited to 'sql/hive')
-rw-r--r--sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala2
1 files changed, 1 insertions, 1 deletions
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
index c48d4ed608..8410a2e4a4 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
@@ -81,7 +81,7 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
options = table.storage.properties)
LogicalRelation(
- dataSource.resolveRelation(checkPathExist = true),
+ dataSource.resolveRelation(),
catalogTable = Some(table))
}
}