diff options
| author | Reynold Xin <rxin@databricks.com> | 2015-01-29 00:01:10 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-01-29 00:01:10 -0800 |
| commit | 5ad78f62056f2560cd371ee964111a646806d0ff (patch) | |
| tree | c5db8104a00b4a835db77bf7f7116622b47c8cc3 /sql/catalyst/src/test | |
| parent | a63be1a18f7b7d77f7deef2abc9a5be6ad24ae28 (diff) | |
| download | spark-5ad78f62056f2560cd371ee964111a646806d0ff.tar.gz spark-5ad78f62056f2560cd371ee964111a646806d0ff.tar.bz2 spark-5ad78f62056f2560cd371ee964111a646806d0ff.zip | |
[SQL] Various DataFrame DSL update.
1. Added foreach, foreachPartition, flatMap to DataFrame.
2. Added col() in dsl.
3. Support renaming columns in toDataFrame.
4. Support type inference on arrays (in addition to Seq).
5. Updated mllib to use the new DSL.
Author: Reynold Xin <rxin@databricks.com>
Closes #4260 from rxin/sql-dsl-update and squashes the following commits:
73466c1 [Reynold Xin] Fixed LogisticRegression. Also added better error message for resolve.
fab3ccc [Reynold Xin] Bug fix.
d31fcd2 [Reynold Xin] Style fix.
62608c4 [Reynold Xin] [SQL] Various DataFrame DSL update.
Diffstat (limited to 'sql/catalyst/src/test')
| -rw-r--r-- | sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala index 5138942a55..4a66716e0a 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala @@ -60,6 +60,7 @@ case class OptionalData( case class ComplexData( arrayField: Seq[Int], + arrayField1: Array[Int], arrayFieldContainsNull: Seq[java.lang.Integer], mapField: Map[Int, Long], mapFieldValueContainsNull: Map[Int, java.lang.Long], @@ -132,6 +133,10 @@ class ScalaReflectionSuite extends FunSuite { ArrayType(IntegerType, containsNull = false), nullable = true), StructField( + "arrayField1", + ArrayType(IntegerType, containsNull = false), + nullable = true), + StructField( "arrayFieldContainsNull", ArrayType(IntegerType, containsNull = true), nullable = true), |