diff options
| author | Liang-Chi Hsieh <simonh@tw.ibm.com> | 2016-05-24 09:43:39 -0700 |
|---|---|---|
| committer | Wenchen Fan <wenchen@databricks.com> | 2016-05-24 09:43:39 -0700 |
| commit | c24b6b679c3efa053f7de19be73eb36dc70d9930 (patch) | |
| tree | 6a0062ba7892812485dcc01b01e731c61e632ca9 /python | |
| parent | 6075f5b4d8e98483d26c31576f58e2229024b4f4 (diff) | |
| download | spark-c24b6b679c3efa053f7de19be73eb36dc70d9930.tar.gz spark-c24b6b679c3efa053f7de19be73eb36dc70d9930.tar.bz2 spark-c24b6b679c3efa053f7de19be73eb36dc70d9930.zip | |
[SPARK-11753][SQL][TEST-HADOOP2.2] Make allowNonNumericNumbers option work
## What changes were proposed in this pull request?
Jackson suppprts `allowNonNumericNumbers` option to parse non-standard non-numeric numbers such as "NaN", "Infinity", "INF". Currently used Jackson version (2.5.3) doesn't support it all. This patch upgrades the library and make the two ignored tests in `JsonParsingOptionsSuite` passed.
## How was this patch tested?
`JsonParsingOptionsSuite`.
Author: Liang-Chi Hsieh <simonh@tw.ibm.com>
Author: Liang-Chi Hsieh <viirya@appier.com>
Closes #9759 from viirya/fix-json-nonnumric.
Diffstat (limited to 'python')
| -rw-r--r-- | python/pyspark/sql/readwriter.py | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py index 855c9d666f..6f788cf50c 100644 --- a/python/pyspark/sql/readwriter.py +++ b/python/pyspark/sql/readwriter.py @@ -193,6 +193,9 @@ class DataFrameReader(object): set, it uses the default value, ``true``. :param allowNumericLeadingZero: allows leading zeros in numbers (e.g. 00012). If None is set, it uses the default value, ``false``. + :param allowNonNumericNumbers: allows using non-numeric numbers such as "NaN", "Infinity", + "-Infinity", "INF", "-INF", which are convertd to floating + point numbers, ``true``. :param allowBackslashEscapingAnyCharacter: allows accepting quoting of all character using backslash quoting mechanism. If None is set, it uses the default value, ``false``. |