diff options
| author | Cheng Hao <hao.cheng@intel.com> | 2014-10-27 20:42:05 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2014-10-27 20:42:05 -0700 |
| commit | 418ad83fe113f2f90552eb7247670279b55aed28 (patch) | |
| tree | ca5a99abff09519e18c17e299938841242b76685 /sql/catalyst | |
| parent | 7e3a1ada86e6adf1ddd4d8a321824daf5f3b2c75 (diff) | |
| download | spark-418ad83fe113f2f90552eb7247670279b55aed28.tar.gz spark-418ad83fe113f2f90552eb7247670279b55aed28.tar.bz2 spark-418ad83fe113f2f90552eb7247670279b55aed28.zip | |
[SPARK-3911] [SQL] HiveSimpleUdf can not be optimized in constant folding
```
explain extended select cos(null) from src limit 1;
```
outputs:
```
Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#5]
MetastoreRelation default, src, None
== Optimized Logical Plan ==
Limit 1
Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#5]
MetastoreRelation default, src, None
== Physical Plan ==
Limit 1
Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#5]
HiveTableScan [], (MetastoreRelation default, src, None), None
```
After patching this PR it outputs
```
== Parsed Logical Plan ==
Limit 1
Project ['cos(null) AS c_0#0]
UnresolvedRelation None, src, None
== Analyzed Logical Plan ==
Limit 1
Project [HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFCos(null) AS c_0#0]
MetastoreRelation default, src, None
== Optimized Logical Plan ==
Limit 1
Project [null AS c_0#0]
MetastoreRelation default, src, None
== Physical Plan ==
Limit 1
Project [null AS c_0#0]
HiveTableScan [], (MetastoreRelation default, src, None), None
```
Author: Cheng Hao <hao.cheng@intel.com>
Closes #2771 from chenghao-intel/hive_udf_constant_folding and squashes the following commits:
1379c73 [Cheng Hao] duplicate the PlanTest with catalyst/plans/PlanTest
1e52dda [Cheng Hao] add unit test for hive simple udf constant folding
01609ff [Cheng Hao] support constant folding for HiveSimpleUdf
Diffstat (limited to 'sql/catalyst')
| -rw-r--r-- | sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala index 7e9f47ef21..c4a1f899d8 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala @@ -33,7 +33,8 @@ class PlanTest extends FunSuite { * we must normalize them to check if two different queries are identical. */ protected def normalizeExprIds(plan: LogicalPlan) = { - val minId = plan.flatMap(_.expressions.flatMap(_.references).map(_.exprId.id)).min + val list = plan.flatMap(_.expressions.flatMap(_.references).map(_.exprId.id)) + val minId = if (list.isEmpty) 0 else list.min plan transformAllExpressions { case a: AttributeReference => AttributeReference(a.name, a.dataType, a.nullable)(exprId = ExprId(a.exprId.id - minId)) |