diff options
| author | Dilip Biswal <dbiswal@us.ibm.com> | 2017-04-04 09:53:05 +0900 |
|---|---|---|
| committer | Takuya UESHIN <ueshin@databricks.com> | 2017-04-04 09:53:05 +0900 |
| commit | 3bfb639cb7352aec572ef6686d3471bd78748ffa (patch) | |
| tree | 9d2054aecd8b3356a4ec7a38543def25a3ee380e /sql/core/src/test/resources/sql-tests/results | |
| parent | e7877fd4728ed41e440d7c4d8b6b02bd0d9e873e (diff) | |
| download | spark-3bfb639cb7352aec572ef6686d3471bd78748ffa.tar.gz spark-3bfb639cb7352aec572ef6686d3471bd78748ffa.tar.bz2 spark-3bfb639cb7352aec572ef6686d3471bd78748ffa.zip | |
[SPARK-10364][SQL] Support Parquet logical type TIMESTAMP_MILLIS
## What changes were proposed in this pull request?
**Description** from JIRA
The TimestampType in Spark SQL is of microsecond precision. Ideally, we should convert Spark SQL timestamp values into Parquet TIMESTAMP_MICROS. But unfortunately parquet-mr hasn't supported it yet.
For the read path, we should be able to read TIMESTAMP_MILLIS Parquet values and pad a 0 microsecond part to read values.
For the write path, currently we are writing timestamps as INT96, similar to Impala and Hive. One alternative is that, we can have a separate SQL option to let users be able to write Spark SQL timestamp values as TIMESTAMP_MILLIS. Of course, in this way the microsecond part will be truncated.
## How was this patch tested?
Added new tests in ParquetQuerySuite and ParquetIOSuite
Author: Dilip Biswal <dbiswal@us.ibm.com>
Closes #15332 from dilipbiswal/parquet-time-millis.
Diffstat (limited to 'sql/core/src/test/resources/sql-tests/results')
0 files changed, 0 insertions, 0 deletions