diff options
| author | Yin Huai <yhuai@databricks.com> | 2016-04-09 23:32:17 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2016-04-09 23:32:17 -0700 |
| commit | 3fb09afd5e55b9a7a0a332273f09f984a78c3645 (patch) | |
| tree | 56745a2d557b9f534808c59e6134775b5c59586b /sql/hive | |
| parent | aea30a1a9b79eb13d362ef32e4e9c8233e29f3dc (diff) | |
| download | spark-3fb09afd5e55b9a7a0a332273f09f984a78c3645.tar.gz spark-3fb09afd5e55b9a7a0a332273f09f984a78c3645.tar.bz2 spark-3fb09afd5e55b9a7a0a332273f09f984a78c3645.zip | |
[SPARK-14506][SQL] HiveClientImpl's toHiveTable misses a table property for external tables
## What changes were proposed in this pull request?
For an external table's metadata (in Hive's representation), its table type needs to be EXTERNAL_TABLE. Also, there needs to be a field called EXTERNAL set in the table property with a value of TRUE (for a MANAGED_TABLE it will be FALSE) based on https://github.com/apache/hive/blob/release-1.2.1/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1095-L1105. HiveClientImpl's toHiveTable misses to set this table property.
## How was this patch tested?
Added a new test.
Author: Yin Huai <yhuai@databricks.com>
Closes #12275 from yhuai/SPARK-14506.
Diffstat (limited to 'sql/hive')
| -rw-r--r-- | sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala | 13 |
1 files changed, 11 insertions, 2 deletions
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala index d0eb9ddf50..a037671ef0 100644 --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala @@ -659,9 +659,18 @@ private[hive] class HiveClientImpl( private def toHiveTable(table: CatalogTable): HiveTable = { val hiveTable = new HiveTable(table.database, table.identifier.table) + // For EXTERNAL_TABLE/MANAGED_TABLE, we also need to set EXTERNAL field in + // the table properties accodringly. Otherwise, if EXTERNAL_TABLE is the table type + // but EXTERNAL field is not set, Hive metastore will change the type to + // MANAGED_TABLE (see + // metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1095-L1105) hiveTable.setTableType(table.tableType match { - case CatalogTableType.EXTERNAL_TABLE => HiveTableType.EXTERNAL_TABLE - case CatalogTableType.MANAGED_TABLE => HiveTableType.MANAGED_TABLE + case CatalogTableType.EXTERNAL_TABLE => + hiveTable.setProperty("EXTERNAL", "TRUE") + HiveTableType.EXTERNAL_TABLE + case CatalogTableType.MANAGED_TABLE => + hiveTable.setProperty("EXTERNAL", "FALSE") + HiveTableType.MANAGED_TABLE case CatalogTableType.INDEX_TABLE => HiveTableType.INDEX_TABLE case CatalogTableType.VIRTUAL_VIEW => HiveTableType.VIRTUAL_VIEW }) |