aboutsummaryrefslogtreecommitdiff
path: root/sql/hive
diff options
context:
space:
mode:
authorhyukjinkwon <gurwls223@gmail.com>2017-04-03 17:44:39 +0800
committerWenchen Fan <wenchen@databricks.com>2017-04-03 17:44:39 +0800
commit4fa1a43af6b5a6abaef7e04cacb2617a2e92d816 (patch)
treec07f8d0c8cdfa086f1734004fb8f11f9526c41d7 /sql/hive
parent4d28e8430d11323f08657ca8f3251ca787c45501 (diff)
downloadspark-4fa1a43af6b5a6abaef7e04cacb2617a2e92d816.tar.gz
spark-4fa1a43af6b5a6abaef7e04cacb2617a2e92d816.tar.bz2
spark-4fa1a43af6b5a6abaef7e04cacb2617a2e92d816.zip
[SPARK-19641][SQL] JSON schema inference in DROPMALFORMED mode produces incorrect schema for non-array/object JSONs
## What changes were proposed in this pull request? Currently, when we infer the types for vaild JSON strings but object or array, we are producing empty schemas regardless of parse modes as below: ```scala scala> spark.read.option("mode", "DROPMALFORMED").json(Seq("""{"a": 1}""", """"a"""").toDS).printSchema() root ``` ```scala scala> spark.read.option("mode", "FAILFAST").json(Seq("""{"a": 1}""", """"a"""").toDS).printSchema() root ``` This PR proposes to handle parse modes in type inference. After this PR, ```scala scala> spark.read.option("mode", "DROPMALFORMED").json(Seq("""{"a": 1}""", """"a"""").toDS).printSchema() root |-- a: long (nullable = true) ``` ``` scala> spark.read.option("mode", "FAILFAST").json(Seq("""{"a": 1}""", """"a"""").toDS).printSchema() java.lang.RuntimeException: Failed to infer a common schema. Struct types are expected but string was found. ``` This PR is based on https://github.com/NathanHowell/spark/commit/e233fd03346a73b3b447fa4c24f3b12c8b2e53ae and I and NathanHowell talked about this in https://issues.apache.org/jira/browse/SPARK-19641 ## How was this patch tested? Unit tests in `JsonSuite` for both `DROPMALFORMED` and `FAILFAST` modes. Author: hyukjinkwon <gurwls223@gmail.com> Closes #17492 from HyukjinKwon/SPARK-19641.
Diffstat (limited to 'sql/hive')
0 files changed, 0 insertions, 0 deletions