aboutsummaryrefslogtreecommitdiff
path: root/sql/core/src/test
diff options
context:
space:
mode:
authorSean Zhong <seanzhong@databricks.com>2016-08-04 19:45:47 +0800
committerWenchen Fan <wenchen@databricks.com>2016-08-04 19:45:47 +0800
commit9d7a47406ed538f0005cdc7a62bc6e6f20634815 (patch)
treeb015623f922bd4646c2e80c9e27fd2349d04a469 /sql/core/src/test
parent43f4fd6f9bfff749af17e3c65b53a33f5ecb0922 (diff)
downloadspark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.tar.gz
spark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.tar.bz2
spark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.zip
[SPARK-16853][SQL] fixes encoder error in DataSet typed select
## What changes were proposed in this pull request? For DataSet typed select: ``` def select[U1: Encoder](c1: TypedColumn[T, U1]): Dataset[U1] ``` If type T is a case class or a tuple class that is not atomic, the resulting logical plan's schema will mismatch with `Dataset[T]` encoder's schema, which will cause encoder error and throw AnalysisException. ### Before change: ``` scala> case class A(a: Int, b: Int) scala> Seq((0, A(1,2))).toDS.select($"_2".as[A]) org.apache.spark.sql.AnalysisException: cannot resolve '`a`' given input columns: [_2]; .. ``` ### After change: ``` scala> case class A(a: Int, b: Int) scala> Seq((0, A(1,2))).toDS.select($"_2".as[A]).show +---+---+ | a| b| +---+---+ | 1| 2| +---+---+ ``` ## How was this patch tested? Unit test. Author: Sean Zhong <seanzhong@databricks.com> Closes #14474 from clockfly/SPARK-16853.
Diffstat (limited to 'sql/core/src/test')
-rw-r--r--sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala11
1 files changed, 11 insertions, 0 deletions
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
index 7e3b7b63d8..8a756fd474 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
@@ -184,6 +184,17 @@ class DatasetSuite extends QueryTest with SharedSQLContext {
2, 3, 4)
}
+ test("SPARK-16853: select, case class and tuple") {
+ val ds = Seq(("a", 1), ("b", 2), ("c", 3)).toDS()
+ checkDataset(
+ ds.select(expr("struct(_2, _2)").as[(Int, Int)]): Dataset[(Int, Int)],
+ (1, 1), (2, 2), (3, 3))
+
+ checkDataset(
+ ds.select(expr("named_struct('a', _1, 'b', _2)").as[ClassData]): Dataset[ClassData],
+ ClassData("a", 1), ClassData("b", 2), ClassData("c", 3))
+ }
+
test("select 2") {
val ds = Seq(("a", 1), ("b", 2), ("c", 3)).toDS()
checkDataset(