diff options
| author | hyukjinkwon <gurwls223@gmail.com> | 2016-10-18 13:20:42 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-10-18 13:20:42 -0700 |
| commit | 37686539f546ac7a3657dbfc59b7ac982b4b9bce (patch) | |
| tree | 907f0d46244012768961aad70c9c998510915b34 /scalastyle-config.xml | |
| parent | e59df62e62ec4c5f8bd02a13f05fa3ec6f0fc694 (diff) | |
| download | spark-37686539f546ac7a3657dbfc59b7ac982b4b9bce.tar.gz spark-37686539f546ac7a3657dbfc59b7ac982b4b9bce.tar.bz2 spark-37686539f546ac7a3657dbfc59b7ac982b4b9bce.zip | |
[SPARK-17388] [SQL] Support for inferring type date/timestamp/decimal for partition column
## What changes were proposed in this pull request?
Currently, Spark only supports to infer `IntegerType`, `LongType`, `DoubleType` and `StringType`.
`DecimalType` is being tried but it seems it never infers type as `DecimalType` as `DoubleType` is being tried first. Also, it seems `DateType` and `TimestampType` could be inferred.
As far as I know, it is pretty common to use both for a partition column.
This PR fixes the incorrect `DecimalType` try and also adds the support for both `DateType` and `TimestampType` for inferring partition column type.
## How was this patch tested?
Unit tests in `ParquetPartitionDiscoverySuite`.
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #14947 from HyukjinKwon/SPARK-17388.
Diffstat (limited to 'scalastyle-config.xml')
0 files changed, 0 insertions, 0 deletions