diff options
author | Cheng Lian <lian@databricks.com> | 2015-10-12 10:17:19 -0700 |
---|---|---|
committer | Davies Liu <davies.liu@gmail.com> | 2015-10-12 10:17:19 -0700 |
commit | 64b1d00e1a7c1dc52c08a5e97baf6e7117f1a94f (patch) | |
tree | 19c8efa0995dca7acb17643c1ad9a3acf152c8b4 /.rat-excludes | |
parent | fcb37a04177edc2376e39dd0b910f0268f7c72ec (diff) | |
download | spark-64b1d00e1a7c1dc52c08a5e97baf6e7117f1a94f.tar.gz spark-64b1d00e1a7c1dc52c08a5e97baf6e7117f1a94f.tar.bz2 spark-64b1d00e1a7c1dc52c08a5e97baf6e7117f1a94f.zip |
[SPARK-11007] [SQL] Adds dictionary aware Parquet decimal converters
For Parquet decimal columns that are encoded using plain-dictionary encoding, we can make the upper level converter aware of the dictionary, so that we can pre-instantiate all the decimals to avoid duplicated instantiation.
Note that plain-dictionary encoding isn't available for `FIXED_LEN_BYTE_ARRAY` for Parquet writer version `PARQUET_1_0`. So currently only decimals written as `INT32` and `INT64` can benefit from this optimization.
Author: Cheng Lian <lian@databricks.com>
Closes #9040 from liancheng/spark-11007.decimal-converter-dict-support.
Diffstat (limited to '.rat-excludes')
0 files changed, 0 insertions, 0 deletions