diff options
| author | Kevin Yu <qyu@us.ibm.com> | 2016-06-09 09:50:09 -0700 |
|---|---|---|
| committer | Wenchen Fan <wenchen@databricks.com> | 2016-06-09 09:50:09 -0700 |
| commit | 99386fe3989f758844de14b2c28eccfdf8163221 (patch) | |
| tree | bc78843acc3dbe4ccc352e7313cdc151a6e61778 /python | |
| parent | 147c020823080c60b495f7950629d8134bf895db (diff) | |
| download | spark-99386fe3989f758844de14b2c28eccfdf8163221.tar.gz spark-99386fe3989f758844de14b2c28eccfdf8163221.tar.bz2 spark-99386fe3989f758844de14b2c28eccfdf8163221.zip | |
[SPARK-15804][SQL] Include metadata in the toStructType
## What changes were proposed in this pull request?
The help function 'toStructType' in the AttributeSeq class doesn't include the metadata when it builds the StructField, so it causes this reported problem https://issues.apache.org/jira/browse/SPARK-15804?jql=project%20%3D%20SPARK when spark writes the the dataframe with the metadata to the parquet datasource.
The code path is when spark writes the dataframe to the parquet datasource through the InsertIntoHadoopFsRelationCommand, spark will build the WriteRelation container, and it will call the help function 'toStructType' to create StructType which contains StructField, it should include the metadata there, otherwise, we will lost the user provide metadata.
## How was this patch tested?
added test case in ParquetQuerySuite.scala
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Author: Kevin Yu <qyu@us.ibm.com>
Closes #13555 from kevinyu98/spark-15804.
Diffstat (limited to 'python')
0 files changed, 0 insertions, 0 deletions