diff options
author | Sean Zhong <seanzhong@databricks.com> | 2016-08-04 19:45:47 +0800 |
---|---|---|
committer | Wenchen Fan <wenchen@databricks.com> | 2016-08-04 19:45:47 +0800 |
commit | 9d7a47406ed538f0005cdc7a62bc6e6f20634815 (patch) | |
tree | b015623f922bd4646c2e80c9e27fd2349d04a469 /sql/catalyst/src/main/scala/org | |
parent | 43f4fd6f9bfff749af17e3c65b53a33f5ecb0922 (diff) | |
download | spark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.tar.gz spark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.tar.bz2 spark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.zip |
[SPARK-16853][SQL] fixes encoder error in DataSet typed select
## What changes were proposed in this pull request?
For DataSet typed select:
```
def select[U1: Encoder](c1: TypedColumn[T, U1]): Dataset[U1]
```
If type T is a case class or a tuple class that is not atomic, the resulting logical plan's schema will mismatch with `Dataset[T]` encoder's schema, which will cause encoder error and throw AnalysisException.
### Before change:
```
scala> case class A(a: Int, b: Int)
scala> Seq((0, A(1,2))).toDS.select($"_2".as[A])
org.apache.spark.sql.AnalysisException: cannot resolve '`a`' given input columns: [_2];
..
```
### After change:
```
scala> case class A(a: Int, b: Int)
scala> Seq((0, A(1,2))).toDS.select($"_2".as[A]).show
+---+---+
| a| b|
+---+---+
| 1| 2|
+---+---+
```
## How was this patch tested?
Unit test.
Author: Sean Zhong <seanzhong@databricks.com>
Closes #14474 from clockfly/SPARK-16853.
Diffstat (limited to 'sql/catalyst/src/main/scala/org')
-rw-r--r-- | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala index 1fac26c438..b96b744b4f 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala @@ -169,6 +169,10 @@ object ExpressionEncoder { ClassTag(cls)) } + // Tuple1 + def tuple[T](e: ExpressionEncoder[T]): ExpressionEncoder[Tuple1[T]] = + tuple(Seq(e)).asInstanceOf[ExpressionEncoder[Tuple1[T]]] + def tuple[T1, T2]( e1: ExpressionEncoder[T1], e2: ExpressionEncoder[T2]): ExpressionEncoder[(T1, T2)] = |