aboutsummaryrefslogtreecommitdiff
path: root/extras/README.md
diff options
context:
space:
mode:
authorTakuya UESHIN <ueshin@happy-camper.st>2014-12-11 22:45:25 -0800
committerMichael Armbrust <michael@databricks.com>2014-12-11 22:45:25 -0800
commit334480362b3a133c2fb1e9af898930fe76d7a163 (patch)
tree240aad85d727d7604a56560b8ec8bffd565419a3 /extras/README.md
parentc152dde78f73d5ce3a483fd60a47e7de1f1916da (diff)
downloadspark-334480362b3a133c2fb1e9af898930fe76d7a163.tar.gz
spark-334480362b3a133c2fb1e9af898930fe76d7a163.tar.bz2
spark-334480362b3a133c2fb1e9af898930fe76d7a163.zip
[SPARK-4293][SQL] Make Cast be able to handle complex types.
Inserting data of type including `ArrayType.containsNull == false` or `MapType.valueContainsNull == false` or `StructType.fields.exists(_.nullable == false)` into Hive table will fail because `Cast` inserted by `HiveMetastoreCatalog.PreInsertionCasts` rule of `Analyzer` can't handle these types correctly. Complex type cast rule proposal: - Cast for non-complex types should be able to cast the same as before. - Cast for `ArrayType` can evaluate if - Element type can cast - Nullability rule doesn't break - Cast for `MapType` can evaluate if - Key type can cast - Nullability for casted key type is `false` - Value type can cast - Nullability rule for value type doesn't break - Cast for `StructType` can evaluate if - The field size is the same - Each field can cast - Nullability rule for each field doesn't break - The nested structure should be the same. Nullability rule: - If the casted type is `nullable == true`, the target nullability should be `true` Author: Takuya UESHIN <ueshin@happy-camper.st> Closes #3150 from ueshin/issues/SPARK-4293 and squashes the following commits: e935939 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-4293 ba14003 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-4293 8999868 [Takuya UESHIN] Fix a test title. f677c30 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-4293 287f410 [Takuya UESHIN] Add tests to insert data of types ArrayType / MapType / StructType with nullability is false into Hive table. 4f71bb8 [Takuya UESHIN] Make Cast be able to handle complex types.
Diffstat (limited to 'extras/README.md')
0 files changed, 0 insertions, 0 deletions