diff options
| author | Dilip Biswal <dbiswal@us.ibm.com> | 2015-12-18 09:54:30 -0800 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2015-12-18 09:54:30 -0800 |
| commit | ee444fe4b8c9f382524e1fa346c67ba6da8104d8 (patch) | |
| tree | c231aae992e2de9bf96e75364d940657c5aa84cb /sql/hive | |
| parent | 2782818287a71925523c1320291db6cb25221e9f (diff) | |
| download | spark-ee444fe4b8c9f382524e1fa346c67ba6da8104d8.tar.gz spark-ee444fe4b8c9f382524e1fa346c67ba6da8104d8.tar.bz2 spark-ee444fe4b8c9f382524e1fa346c67ba6da8104d8.zip | |
[SPARK-11619][SQL] cannot use UDTF in DataFrame.selectExpr
Description of the problem from cloud-fan
Actually this line: https://github.com/apache/spark/blob/branch-1.5/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala#L689
When we use `selectExpr`, we pass in `UnresolvedFunction` to `DataFrame.select` and fall in the last case. A workaround is to do special handling for UDTF like we did for `explode`(and `json_tuple` in 1.6), wrap it with `MultiAlias`.
Another workaround is using `expr`, for example, `df.select(expr("explode(a)").as(Nil))`, I think `selectExpr` is no longer needed after we have the `expr` function....
Author: Dilip Biswal <dbiswal@us.ibm.com>
Closes #9981 from dilipbiswal/spark-11619.
Diffstat (limited to 'sql/hive')
| -rw-r--r-- | sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala index da41b659e3..0e89928cb6 100644 --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala @@ -1107,7 +1107,7 @@ https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C // (if there is a group by) or a script transformation. val withProject: LogicalPlan = transformation.getOrElse { val selectExpressions = - select.getChildren.asScala.flatMap(selExprNodeToExpr).map(UnresolvedAlias) + select.getChildren.asScala.flatMap(selExprNodeToExpr).map(UnresolvedAlias(_)) Seq( groupByClause.map(e => e match { case Token("TOK_GROUPBY", children) => |