diff options
| author | Cheng Lian <lian@databricks.com> | 2015-12-09 23:30:42 +0800 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-12-09 23:30:42 +0800 |
| commit | 6e1c55eac4849669e119ce0d51f6d051830deb9f (patch) | |
| tree | 4d8286eb371c2de27cc374ed55749c2830add681 /sql/hive | |
| parent | a113216865fd45ea39ae8f104e784af2cf667dcf (diff) | |
| download | spark-6e1c55eac4849669e119ce0d51f6d051830deb9f.tar.gz spark-6e1c55eac4849669e119ce0d51f6d051830deb9f.tar.bz2 spark-6e1c55eac4849669e119ce0d51f6d051830deb9f.zip | |
[SPARK-12012][SQL] Show more comprehensive PhysicalRDD metadata when visualizing SQL query plan
This PR adds a `private[sql]` method `metadata` to `SparkPlan`, which can be used to describe detail information about a physical plan during visualization. Specifically, this PR uses this method to provide details of `PhysicalRDD`s translated from a data source relation. For example, a `ParquetRelation` converted from Hive metastore table `default.psrc` is now shown as the following screenshot:
And here is the screenshot for a regular `ParquetRelation` (not converted from Hive metastore table) loaded from a really long path:
Author: Cheng Lian <lian@databricks.com>
Closes #10004 from liancheng/spark-12012.physical-rdd-metadata.
Diffstat (limited to 'sql/hive')
| -rw-r--r-- | sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala | 7 |
1 files changed, 6 insertions, 1 deletions
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala index 9a981d02ad..08b291e088 100644 --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala @@ -411,7 +411,12 @@ private[hive] class HiveMetastoreCatalog(val client: ClientInterface, hive: Hive // evil case insensitivity issue, which is reconciled within `ParquetRelation`. val parquetOptions = Map( ParquetRelation.METASTORE_SCHEMA -> metastoreSchema.json, - ParquetRelation.MERGE_SCHEMA -> mergeSchema.toString) + ParquetRelation.MERGE_SCHEMA -> mergeSchema.toString, + ParquetRelation.METASTORE_TABLE_NAME -> TableIdentifier( + metastoreRelation.tableName, + Some(metastoreRelation.databaseName) + ).unquotedString + ) val tableIdentifier = QualifiedTableName(metastoreRelation.databaseName, metastoreRelation.tableName) |