diff options
| author | Daoyuan Wang <daoyuan.wang@intel.com> | 2015-02-03 12:06:06 -0800 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2015-02-03 12:06:06 -0800 |
| commit | 0c20ce69fb4bcb1cec5313a9d072826c5588cbbc (patch) | |
| tree | 5aeb2c2c72403440a6dd3bef398fbce92d5a5d7f /docs | |
| parent | 4204a1271d5bff4dd64f46eed9ee80b30081f9dc (diff) | |
| download | spark-0c20ce69fb4bcb1cec5313a9d072826c5588cbbc.tar.gz spark-0c20ce69fb4bcb1cec5313a9d072826c5588cbbc.tar.bz2 spark-0c20ce69fb4bcb1cec5313a9d072826c5588cbbc.zip | |
[SPARK-4987] [SQL] parquet timestamp type support
Author: Daoyuan Wang <daoyuan.wang@intel.com>
Closes #3820 from adrian-wang/parquettimestamp and squashes the following commits:
b1e2a0d [Daoyuan Wang] fix for nanos
4dadef1 [Daoyuan Wang] fix wrong read
93f438d [Daoyuan Wang] parquet timestamp support
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/sql-programming-guide.md | 9 |
1 files changed, 9 insertions, 0 deletions
diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index be8c5c2c15..22664b419f 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -581,6 +581,15 @@ Configuration of Parquet can be done using the `setConf` method on SQLContext or </td> </tr> <tr> + <td><code>spark.sql.parquet.int96AsTimestamp</code></td> + <td>true</td> + <td> + Some Parquet-producing systems, in particular Impala, store Timestamp into INT96. Spark would also + store Timestamp as INT96 because we need to avoid precision lost of the nanoseconds field. This + flag tells Spark SQL to interpret INT96 data as a timestamp to provide compatibility with these systems. + </td> +</tr> +<tr> <td><code>spark.sql.parquet.cacheMetadata</code></td> <td>true</td> <td> |