aboutsummaryrefslogtreecommitdiff
path: root/sql/catalyst/src/main
diff options
context:
space:
mode:
authorSean Zhong <seanzhong@databricks.com>2016-08-04 19:45:47 +0800
committerWenchen Fan <wenchen@databricks.com>2016-08-04 19:45:47 +0800
commit9d7a47406ed538f0005cdc7a62bc6e6f20634815 (patch)
treeb015623f922bd4646c2e80c9e27fd2349d04a469 /sql/catalyst/src/main
parent43f4fd6f9bfff749af17e3c65b53a33f5ecb0922 (diff)
downloadspark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.tar.gz
spark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.tar.bz2
spark-9d7a47406ed538f0005cdc7a62bc6e6f20634815.zip
[SPARK-16853][SQL] fixes encoder error in DataSet typed select
## What changes were proposed in this pull request? For DataSet typed select: ``` def select[U1: Encoder](c1: TypedColumn[T, U1]): Dataset[U1] ``` If type T is a case class or a tuple class that is not atomic, the resulting logical plan's schema will mismatch with `Dataset[T]` encoder's schema, which will cause encoder error and throw AnalysisException. ### Before change: ``` scala> case class A(a: Int, b: Int) scala> Seq((0, A(1,2))).toDS.select($"_2".as[A]) org.apache.spark.sql.AnalysisException: cannot resolve '`a`' given input columns: [_2]; .. ``` ### After change: ``` scala> case class A(a: Int, b: Int) scala> Seq((0, A(1,2))).toDS.select($"_2".as[A]).show +---+---+ | a| b| +---+---+ | 1| 2| +---+---+ ``` ## How was this patch tested? Unit test. Author: Sean Zhong <seanzhong@databricks.com> Closes #14474 from clockfly/SPARK-16853.
Diffstat (limited to 'sql/catalyst/src/main')
-rw-r--r--sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala4
1 files changed, 4 insertions, 0 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala
index 1fac26c438..b96b744b4f 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala
@@ -169,6 +169,10 @@ object ExpressionEncoder {
ClassTag(cls))
}
+ // Tuple1
+ def tuple[T](e: ExpressionEncoder[T]): ExpressionEncoder[Tuple1[T]] =
+ tuple(Seq(e)).asInstanceOf[ExpressionEncoder[Tuple1[T]]]
+
def tuple[T1, T2](
e1: ExpressionEncoder[T1],
e2: ExpressionEncoder[T2]): ExpressionEncoder[(T1, T2)] =