aboutsummaryrefslogtreecommitdiff
path: root/sql/core
diff options
context:
space:
mode:
authorDongjoon Hyun <dongjoon@apache.org>2016-05-19 15:57:44 +0800
committerWenchen Fan <wenchen@databricks.com>2016-05-19 15:57:44 +0800
commit5907ebfc11aac8029cfc6d5f8e91cd5f53af54fe (patch)
treecb12d20bcba3d75165da3b04ad96717f9a8d1f92 /sql/core
parente2ec32dab8530aa21ec95a27d60b1c22f3d1a18c (diff)
downloadspark-5907ebfc11aac8029cfc6d5f8e91cd5f53af54fe.tar.gz
spark-5907ebfc11aac8029cfc6d5f8e91cd5f53af54fe.tar.bz2
spark-5907ebfc11aac8029cfc6d5f8e91cd5f53af54fe.zip
[SPARK-14939][SQL] Add FoldablePropagation optimizer
## What changes were proposed in this pull request? This PR aims to add new **FoldablePropagation** optimizer that propagates foldable expressions by replacing all attributes with the aliases of original foldable expression. Other optimizations will take advantage of the propagated foldable expressions: e.g. `EliminateSorts` optimizer now can handle the following Case 2 and 3. (Case 1 is the previous implementation.) 1. Literals and foldable expression, e.g. "ORDER BY 1.0, 'abc', Now()" 2. Foldable ordinals, e.g. "SELECT 1.0, 'abc', Now() ORDER BY 1, 2, 3" 3. Foldable aliases, e.g. "SELECT 1.0 x, 'abc' y, Now() z ORDER BY x, y, z" This PR has been generalized based on cloud-fan 's key ideas many times; he should be credited for the work he did. **Before** ``` scala> sql("SELECT 1.0, Now() x ORDER BY 1, x").explain == Physical Plan == WholeStageCodegen : +- Sort [1.0#5 ASC,x#0 ASC], true, 0 : +- INPUT +- Exchange rangepartitioning(1.0#5 ASC, x#0 ASC, 200), None +- WholeStageCodegen : +- Project [1.0 AS 1.0#5,1461873043577000 AS x#0] : +- INPUT +- Scan OneRowRelation[] ``` **After** ``` scala> sql("SELECT 1.0, Now() x ORDER BY 1, x").explain == Physical Plan == WholeStageCodegen : +- Project [1.0 AS 1.0#5,1461873079484000 AS x#0] : +- INPUT +- Scan OneRowRelation[] ``` ## How was this patch tested? Pass the Jenkins tests including a new test case. Author: Dongjoon Hyun <dongjoon@apache.org> Closes #12719 from dongjoon-hyun/SPARK-14939.
Diffstat (limited to 'sql/core')
-rw-r--r--sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala8
1 files changed, 8 insertions, 0 deletions
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
index 010dea5b30..743a27aa7a 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
@@ -2499,6 +2499,14 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
}
}
+ test("Eliminate noop ordinal ORDER BY") {
+ withSQLConf(SQLConf.ORDER_BY_ORDINAL.key -> "true") {
+ val plan1 = sql("SELECT 1.0, 'abc', year(current_date()) ORDER BY 1, 2, 3")
+ val plan2 = sql("SELECT 1.0, 'abc', year(current_date())")
+ comparePlans(plan1.queryExecution.optimizedPlan, plan2.queryExecution.optimizedPlan)
+ }
+ }
+
test("check code injection is prevented") {
// The end of comment (*/) should be escaped.
var literal =