diff options
| author | Cheng Lian <lian@databricks.com> | 2016-06-23 23:11:46 -0700 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2016-06-23 23:11:46 -0700 |
| commit | 2d2f607bfae97f2681df24f48bb8b1b483c6b309 (patch) | |
| tree | 917059d4730787edb05f1f5d6387ce5de0373d5e /repl/scala-2.10/src | |
| parent | cc6778ee0bf4fa7a78abd30542c4a6f80ea371c5 (diff) | |
| download | spark-2d2f607bfae97f2681df24f48bb8b1b483c6b309.tar.gz spark-2d2f607bfae97f2681df24f48bb8b1b483c6b309.tar.bz2 spark-2d2f607bfae97f2681df24f48bb8b1b483c6b309.zip | |
[SPARK-13709][SQL] Initialize deserializer with both table and partition properties when reading partitioned tables
## What changes were proposed in this pull request?
When reading partitions of a partitioned Hive SerDe table, we only initializes the deserializer using partition properties. However, for SerDes like `AvroSerDe`, essential properties (e.g. Avro schema information) may be defined in table properties. We should merge both table properties and partition properties before initializing the deserializer.
Note that an individual partition may have different properties than the one defined in the table properties (e.g. partitions within a table can have different SerDes). Thus, for any property key defined in both partition and table properties, the value set in partition properties wins.
## How was this patch tested?
New test case added in `QueryPartitionSuite`.
Author: Cheng Lian <lian@databricks.com>
Closes #13865 from liancheng/spark-13709-partitioned-avro-table.
Diffstat (limited to 'repl/scala-2.10/src')
0 files changed, 0 insertions, 0 deletions