diff options
| author | Takuya UESHIN <ueshin@happy-camper.st> | 2014-07-14 15:42:28 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2014-07-14 15:42:35 -0700 |
| commit | 9fe693b5b6ed6af34ee1e800ab89c8a11991ea38 (patch) | |
| tree | 08dbe266642d4754e879e0e91d3f7fa1f9adaf30 /project | |
| parent | 3dd8af7a6623201c28231f4b71f59ea4e9ae29bf (diff) | |
| download | spark-9fe693b5b6ed6af34ee1e800ab89c8a11991ea38.tar.gz spark-9fe693b5b6ed6af34ee1e800ab89c8a11991ea38.tar.bz2 spark-9fe693b5b6ed6af34ee1e800ab89c8a11991ea38.zip | |
[SPARK-2446][SQL] Add BinaryType support to Parquet I/O.
Note that this commit changes the semantics when loading in data that was created with prior versions of Spark SQL. Before, we were writing out strings as Binary data without adding any other annotations. Thus, when data is read in from prior versions, data that was StringType will now become BinaryType. Users that need strings can CAST that column to a String. It was decided that while this breaks compatibility, it does make us compatible with other systems (Hive, Thrift, etc) and adds support for Binary data, so this is the right decision long term.
To support `BinaryType`, the following changes are needed:
- Make `StringType` use `OriginalType.UTF8`
- Add `BinaryType` using `PrimitiveTypeName.BINARY` without `OriginalType`
Author: Takuya UESHIN <ueshin@happy-camper.st>
Closes #1373 from ueshin/issues/SPARK-2446 and squashes the following commits:
ecacb92 [Takuya UESHIN] Add BinaryType support to Parquet I/O.
616e04a [Takuya UESHIN] Make StringType use OriginalType.UTF8.
Diffstat (limited to 'project')
0 files changed, 0 insertions, 0 deletions