aboutsummaryrefslogtreecommitdiff
path: root/sql/hive/src/test/scala/org
diff options
context:
space:
mode:
authorCheng Lian <lian@databricks.com>2017-03-10 15:19:32 -0800
committerWenchen Fan <wenchen@databricks.com>2017-03-10 15:19:32 -0800
commitffee4f1cefb0dfd8d9145ee3be82c6f7b799870b (patch)
tree5f264abe401434c141e99463115c7ad44d88e50d /sql/hive/src/test/scala/org
parentbc30351404d8bc610cbae65fdc12ca613e7735c6 (diff)
downloadspark-ffee4f1cefb0dfd8d9145ee3be82c6f7b799870b.tar.gz
spark-ffee4f1cefb0dfd8d9145ee3be82c6f7b799870b.tar.bz2
spark-ffee4f1cefb0dfd8d9145ee3be82c6f7b799870b.zip
[SPARK-19905][SQL] Bring back Dataset.inputFiles for Hive SerDe tables
## What changes were proposed in this pull request? `Dataset.inputFiles` works by matching `FileRelation`s in the query plan. In Spark 2.1, Hive SerDe tables are represented by `MetastoreRelation`, which inherits from `FileRelation`. However, in Spark 2.2, Hive SerDe tables are now represented by `CatalogRelation`, which doesn't inherit from `FileRelation` anymore, due to the unification of Hive SerDe tables and data source tables. This change breaks `Dataset.inputFiles` for Hive SerDe tables. This PR tries to fix this issue by explicitly matching `CatalogRelation`s that are Hive SerDe tables in `Dataset.inputFiles`. Note that we can't make `CatalogRelation` inherit from `FileRelation` since not all `CatalogRelation`s are file based (e.g., JDBC data source tables). ## How was this patch tested? New test case added in `HiveDDLSuite`. Author: Cheng Lian <lian@databricks.com> Closes #17247 from liancheng/spark-19905-hive-table-input-files.
Diffstat (limited to 'sql/hive/src/test/scala/org')
-rw-r--r--sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala11
1 files changed, 11 insertions, 0 deletions
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
index 23aea24697..79ad156c55 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
@@ -1865,4 +1865,15 @@ class HiveDDLSuite
}
}
}
+
+ test("SPARK-19905: Hive SerDe table input paths") {
+ withTable("spark_19905") {
+ withTempView("spark_19905_view") {
+ spark.range(10).createOrReplaceTempView("spark_19905_view")
+ sql("CREATE TABLE spark_19905 STORED AS RCFILE AS SELECT * FROM spark_19905_view")
+ assert(spark.table("spark_19905").inputFiles.nonEmpty)
+ assert(sql("SELECT input_file_name() FROM spark_19905").count() > 0)
+ }
+ }
+ }
}