diff options
author | Sean Zhong <seanzhong@databricks.com> | 2016-08-04 19:45:47 +0800 |
---|---|---|
committer | Wenchen Fan <wenchen@databricks.com> | 2016-08-04 19:45:47 +0800 |
commit | 9d7a47406ed538f0005cdc7a62bc6e6f20634815 (patch) | |
tree | b015623f922bd4646c2e80c9e27fd2349d04a469 /sql/core/src/test | |
parent | 43f4fd6f9bfff749af17e3c65b53a33f5ecb0922 (diff) | |
download | spark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.tar.gz spark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.tar.bz2 spark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.zip |
[SPARK-16853][SQL] fixes encoder error in DataSet typed select
## What changes were proposed in this pull request?
For DataSet typed select:
```
def select[U1: Encoder](c1: TypedColumn[T, U1]): Dataset[U1]
```
If type T is a case class or a tuple class that is not atomic, the resulting logical plan's schema will mismatch with `Dataset[T]` encoder's schema, which will cause encoder error and throw AnalysisException.
### Before change:
```
scala> case class A(a: Int, b: Int)
scala> Seq((0, A(1,2))).toDS.select($"_2".as[A])
org.apache.spark.sql.AnalysisException: cannot resolve '`a`' given input columns: [_2];
..
```
### After change:
```
scala> case class A(a: Int, b: Int)
scala> Seq((0, A(1,2))).toDS.select($"_2".as[A]).show
+---+---+
| a| b|
+---+---+
| 1| 2|
+---+---+
```
## How was this patch tested?
Unit test.
Author: Sean Zhong <seanzhong@databricks.com>
Closes #14474 from clockfly/SPARK-16853.
Diffstat (limited to 'sql/core/src/test')
-rw-r--r-- | sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala | 11 |
1 files changed, 11 insertions, 0 deletions
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala index 7e3b7b63d8..8a756fd474 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala @@ -184,6 +184,17 @@ class DatasetSuite extends QueryTest with SharedSQLContext { 2, 3, 4) } + test("SPARK-16853: select, case class and tuple") { + val ds = Seq(("a", 1), ("b", 2), ("c", 3)).toDS() + checkDataset( + ds.select(expr("struct(_2, _2)").as[(Int, Int)]): Dataset[(Int, Int)], + (1, 1), (2, 2), (3, 3)) + + checkDataset( + ds.select(expr("named_struct('a', _1, 'b', _2)").as[ClassData]): Dataset[ClassData], + ClassData("a", 1), ClassData("b", 2), ClassData("c", 3)) + } + test("select 2") { val ds = Seq(("a", 1), ("b", 2), ("c", 3)).toDS() checkDataset( |