diff options
| author | Michael Armbrust <michael@databricks.com> | 2014-05-05 22:59:42 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-05-05 22:59:42 -0700 |
| commit | 3c64750bdd4c2d0a5562f90aead37be81627cc9d (patch) | |
| tree | a163bcb9f71d50f21ba3f233418faa55305ebfb0 /sql/catalyst | |
| parent | a2262cdb7aa30e9f45043f1440d4b02bc3340f9f (diff) | |
| download | spark-3c64750bdd4c2d0a5562f90aead37be81627cc9d.tar.gz spark-3c64750bdd4c2d0a5562f90aead37be81627cc9d.tar.bz2 spark-3c64750bdd4c2d0a5562f90aead37be81627cc9d.zip | |
[SQL] SPARK-1732 - Support for null primitive values.
I also removed a println that I bumped into.
Author: Michael Armbrust <michael@databricks.com>
Closes #658 from marmbrus/nullPrimitives and squashes the following commits:
a3ec4f3 [Michael Armbrust] Remove println.
695606b [Michael Armbrust] Support for null primatives from using scala and java reflection.
Diffstat (limited to 'sql/catalyst')
| -rw-r--r-- | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala | 14 |
1 files changed, 13 insertions, 1 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala index 446d0e0bd7..792ef6cee6 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala @@ -44,7 +44,8 @@ object ScalaReflection { case t if t <:< typeOf[Product] => val params = t.member("<init>": TermName).asMethod.paramss StructType( - params.head.map(p => StructField(p.name.toString, schemaFor(p.typeSignature), true))) + params.head.map(p => + StructField(p.name.toString, schemaFor(p.typeSignature), nullable = true))) // Need to decide if we actually need a special type here. case t if t <:< typeOf[Array[Byte]] => BinaryType case t if t <:< typeOf[Array[_]] => @@ -58,6 +59,17 @@ object ScalaReflection { case t if t <:< typeOf[String] => StringType case t if t <:< typeOf[Timestamp] => TimestampType case t if t <:< typeOf[BigDecimal] => DecimalType + case t if t <:< typeOf[Option[_]] => + val TypeRef(_, _, Seq(optType)) = t + schemaFor(optType) + case t if t <:< typeOf[java.lang.Integer] => IntegerType + case t if t <:< typeOf[java.lang.Long] => LongType + case t if t <:< typeOf[java.lang.Double] => DoubleType + case t if t <:< typeOf[java.lang.Float] => FloatType + case t if t <:< typeOf[java.lang.Short] => ShortType + case t if t <:< typeOf[java.lang.Byte] => ByteType + case t if t <:< typeOf[java.lang.Boolean] => BooleanType + // TODO: The following datatypes could be marked as non-nullable. case t if t <:< definitions.IntTpe => IntegerType case t if t <:< definitions.LongTpe => LongType case t if t <:< definitions.DoubleTpe => DoubleType |