diff options
Diffstat (limited to 'python/pyspark/sql')
| -rw-r--r-- | python/pyspark/sql/readwriter.py | 5 |
1 files changed, 4 insertions, 1 deletions
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py index bae9e69df8..cca57a385c 100644 --- a/python/pyspark/sql/readwriter.py +++ b/python/pyspark/sql/readwriter.py @@ -166,10 +166,13 @@ class DataFrameReader(object): during parsing. * ``PERMISSIVE`` : sets other fields to ``null`` when it meets a corrupted \ record and puts the malformed string into a new field configured by \ - ``spark.sql.columnNameOfCorruptRecord``. When a schema is set by user, it sets \ + ``columnNameOfCorruptRecord``. When a schema is set by user, it sets \ ``null`` for extra fields. * ``DROPMALFORMED`` : ignores the whole corrupted records. * ``FAILFAST`` : throws an exception when it meets corrupted records. + * ``columnNameOfCorruptRecord`` (default ``_corrupt_record``): allows renaming the \ + new field having malformed string created by ``PERMISSIVE`` mode. \ + This overrides ``spark.sql.columnNameOfCorruptRecord``. >>> df1 = sqlContext.read.json('python/test_support/sql/people.json') >>> df1.dtypes |