diff options
| author | Takuya UESHIN <ueshin@happy-camper.st> | 2014-08-26 13:22:55 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2014-08-26 13:22:55 -0700 |
| commit | 98c2bb0bbde6fb2b6f64af3efffefcb0dae94c12 (patch) | |
| tree | ded21f0b71756a5d03c9c77cad09f90fadd69d20 /python/pyspark | |
| parent | 3cedc4f4d78e093fd362085e0a077bb9e4f28ca5 (diff) | |
| download | spark-98c2bb0bbde6fb2b6f64af3efffefcb0dae94c12.tar.gz spark-98c2bb0bbde6fb2b6f64af3efffefcb0dae94c12.tar.bz2 spark-98c2bb0bbde6fb2b6f64af3efffefcb0dae94c12.zip | |
[SPARK-2969][SQL] Make ScalaReflection be able to handle ArrayType.containsNull and MapType.valueContainsNull.
Make `ScalaReflection` be able to handle like:
- `Seq[Int]` as `ArrayType(IntegerType, containsNull = false)`
- `Seq[java.lang.Integer]` as `ArrayType(IntegerType, containsNull = true)`
- `Map[Int, Long]` as `MapType(IntegerType, LongType, valueContainsNull = false)`
- `Map[Int, java.lang.Long]` as `MapType(IntegerType, LongType, valueContainsNull = true)`
Author: Takuya UESHIN <ueshin@happy-camper.st>
Closes #1889 from ueshin/issues/SPARK-2969 and squashes the following commits:
24f1c5c [Takuya UESHIN] Change the default value of ArrayType.containsNull to true in Python API.
79f5b65 [Takuya UESHIN] Change the default value of ArrayType.containsNull to true in Java API.
7cd1a7a [Takuya UESHIN] Fix json test failures.
2cfb862 [Takuya UESHIN] Change the default value of ArrayType.containsNull to true.
2f38e61 [Takuya UESHIN] Revert the default value of MapTypes.valueContainsNull.
9fa02f5 [Takuya UESHIN] Fix a test failure.
1a9a96b [Takuya UESHIN] Modify ScalaReflection to handle ArrayType.containsNull and MapType.valueContainsNull.
Diffstat (limited to 'python/pyspark')
| -rw-r--r-- | python/pyspark/sql.py | 6 |
1 files changed, 3 insertions, 3 deletions
diff --git a/python/pyspark/sql.py b/python/pyspark/sql.py index d4ca0cc8f3..0ff6a548a8 100644 --- a/python/pyspark/sql.py +++ b/python/pyspark/sql.py @@ -186,15 +186,15 @@ class ArrayType(DataType): """ - def __init__(self, elementType, containsNull=False): + def __init__(self, elementType, containsNull=True): """Creates an ArrayType :param elementType: the data type of elements. :param containsNull: indicates whether the list contains None values. - >>> ArrayType(StringType) == ArrayType(StringType, False) + >>> ArrayType(StringType) == ArrayType(StringType, True) True - >>> ArrayType(StringType, True) == ArrayType(StringType) + >>> ArrayType(StringType, False) == ArrayType(StringType) False """ self.elementType = elementType |