diff options
author | Davies Liu <davies.liu@gmail.com> | 2014-08-06 11:08:12 -0700 |
---|---|---|
committer | Michael Armbrust <michael@databricks.com> | 2014-08-06 11:08:12 -0700 |
commit | 48789117c2dd6d38e0bd8d21cdbcb989913205a6 (patch) | |
tree | 387fbb2c70cd2abe993e6009157dd76d6afd0f60 /python/pyspark/sql.py | |
parent | 09f7e4587bbdf74207d2629e8c1314f93d865999 (diff) | |
download | spark-48789117c2dd6d38e0bd8d21cdbcb989913205a6.tar.gz spark-48789117c2dd6d38e0bd8d21cdbcb989913205a6.tar.bz2 spark-48789117c2dd6d38e0bd8d21cdbcb989913205a6.zip |
[SPARK-2875] [PySpark] [SQL] handle null in schemaRDD()
Handle null in schemaRDD during converting them into Python.
Author: Davies Liu <davies.liu@gmail.com>
Closes #1802 from davies/json and squashes the following commits:
88e6b1f [Davies Liu] handle null in schemaRDD()
Diffstat (limited to 'python/pyspark/sql.py')
-rw-r--r-- | python/pyspark/sql.py | 7 |
1 files changed, 7 insertions, 0 deletions
diff --git a/python/pyspark/sql.py b/python/pyspark/sql.py index f1093701dd..adc56e7ec0 100644 --- a/python/pyspark/sql.py +++ b/python/pyspark/sql.py @@ -1231,6 +1231,13 @@ class SQLContext: ... "field3.field5[0] as f3 from table3") >>> srdd6.collect() [Row(f1=u'row1', f2=None,...Row(f1=u'row3', f2=[], f3=None)] + + >>> sqlCtx.jsonRDD(sc.parallelize(['{}', + ... '{"key0": {"key1": "value1"}}'])).collect() + [Row(key0=None), Row(key0=Row(key1=u'value1'))] + >>> sqlCtx.jsonRDD(sc.parallelize(['{"key0": null}', + ... '{"key0": {"key1": "value1"}}'])).collect() + [Row(key0=None), Row(key0=Row(key1=u'value1'))] """ def func(iterator): |