diff options
author | Yin Huai <huaiyin.thu@gmail.com> | 2014-07-07 17:05:59 -0700 |
---|---|---|
committer | Michael Armbrust <michael@databricks.com> | 2014-07-07 17:05:59 -0700 |
commit | f0496ee10847db921a028a34f70385f9b740b3f3 (patch) | |
tree | e7ce39f5c6c6c7ad11fe7a904ee488682430e387 /python/pyspark/storagelevel.py | |
parent | 4deeed17c4847f212a4fa1a8685cfe8a12179263 (diff) | |
download | spark-f0496ee10847db921a028a34f70385f9b740b3f3.tar.gz spark-f0496ee10847db921a028a34f70385f9b740b3f3.tar.bz2 spark-f0496ee10847db921a028a34f70385f9b740b3f3.zip |
[SPARK-2375][SQL] JSON schema inference may not resolve type conflicts correctly for a field inside an array of structs
For example, for
```
{"array": [{"field":214748364700}, {"field":1}]}
```
the type of field is resolved as IntType. While, for
```
{"array": [{"field":1}, {"field":214748364700}]}
```
the type of field is resolved as LongType.
JIRA: https://issues.apache.org/jira/browse/SPARK-2375
Author: Yin Huai <huaiyin.thu@gmail.com>
Closes #1308 from yhuai/SPARK-2375 and squashes the following commits:
3e2e312 [Yin Huai] Update unit test.
1b2ff9f [Yin Huai] Merge remote-tracking branch 'upstream/master' into SPARK-2375
10794eb [Yin Huai] Correctly resolve the type of a field inside an array of structs.
Diffstat (limited to 'python/pyspark/storagelevel.py')
0 files changed, 0 insertions, 0 deletions