diff options
author | Takuya UESHIN <ueshin@happy-camper.st> | 2014-12-11 22:45:25 -0800 |
---|---|---|
committer | Michael Armbrust <michael@databricks.com> | 2014-12-11 22:45:25 -0800 |
commit | 334480362b3a133c2fb1e9af898930fe76d7a163 (patch) | |
tree | 240aad85d727d7604a56560b8ec8bffd565419a3 /extras/README.md | |
parent | c152dde78f73d5ce3a483fd60a47e7de1f1916da (diff) | |
download | spark-334480362b3a133c2fb1e9af898930fe76d7a163.tar.gz spark-334480362b3a133c2fb1e9af898930fe76d7a163.tar.bz2 spark-334480362b3a133c2fb1e9af898930fe76d7a163.zip |
[SPARK-4293][SQL] Make Cast be able to handle complex types.
Inserting data of type including `ArrayType.containsNull == false` or `MapType.valueContainsNull == false` or `StructType.fields.exists(_.nullable == false)` into Hive table will fail because `Cast` inserted by `HiveMetastoreCatalog.PreInsertionCasts` rule of `Analyzer` can't handle these types correctly.
Complex type cast rule proposal:
- Cast for non-complex types should be able to cast the same as before.
- Cast for `ArrayType` can evaluate if
- Element type can cast
- Nullability rule doesn't break
- Cast for `MapType` can evaluate if
- Key type can cast
- Nullability for casted key type is `false`
- Value type can cast
- Nullability rule for value type doesn't break
- Cast for `StructType` can evaluate if
- The field size is the same
- Each field can cast
- Nullability rule for each field doesn't break
- The nested structure should be the same.
Nullability rule:
- If the casted type is `nullable == true`, the target nullability should be `true`
Author: Takuya UESHIN <ueshin@happy-camper.st>
Closes #3150 from ueshin/issues/SPARK-4293 and squashes the following commits:
e935939 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-4293
ba14003 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-4293
8999868 [Takuya UESHIN] Fix a test title.
f677c30 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-4293
287f410 [Takuya UESHIN] Add tests to insert data of types ArrayType / MapType / StructType with nullability is false into Hive table.
4f71bb8 [Takuya UESHIN] Make Cast be able to handle complex types.
Diffstat (limited to 'extras/README.md')
0 files changed, 0 insertions, 0 deletions