diff options
| author | Davies Liu <davies@databricks.com> | 2015-08-25 16:00:44 +0800 |
|---|---|---|
| committer | Cheng Lian <lian@databricks.com> | 2015-08-25 16:00:44 +0800 |
| commit | 2f493f7e3924b769160a16f73cccbebf21973b91 (patch) | |
| tree | b431868139f28ea2e6560614118491c750741f12 /sql/hive | |
| parent | 1fc37581a52530bac5d555dbf14927a5780c3b75 (diff) | |
| download | spark-2f493f7e3924b769160a16f73cccbebf21973b91.tar.gz spark-2f493f7e3924b769160a16f73cccbebf21973b91.tar.bz2 spark-2f493f7e3924b769160a16f73cccbebf21973b91.zip | |
[SPARK-10177] [SQL] fix reading Timestamp in parquet from Hive
We misunderstood the Julian days and nanoseconds of the day in parquet (as TimestampType) from Hive/Impala, they are overlapped, so can't be added together directly.
In order to avoid the confusing rounding when do the converting, we use `2440588` as the Julian Day of epoch of unix timestamp (which should be 2440587.5).
Author: Davies Liu <davies@databricks.com>
Author: Cheng Lian <lian@databricks.com>
Closes #8400 from davies/timestamp_parquet.
Diffstat (limited to 'sql/hive')
| -rw-r--r-- | sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetHiveCompatibilitySuite.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetHiveCompatibilitySuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetHiveCompatibilitySuite.scala index bc30180cf0..91d7a48208 100644 --- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetHiveCompatibilitySuite.scala +++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetHiveCompatibilitySuite.scala @@ -113,7 +113,7 @@ class ParquetHiveCompatibilitySuite extends ParquetCompatibilityTest with Before "BOOLEAN", "TINYINT", "SMALLINT", "INT", "BIGINT", "FLOAT", "DOUBLE", "STRING") } - ignore("SPARK-10177 timestamp") { + test("SPARK-10177 timestamp") { testParquetHiveCompatibility(Row(Timestamp.valueOf("2015-08-24 00:31:00")), "TIMESTAMP") } |