aboutsummaryrefslogtreecommitdiff
path: root/sql/catalyst/src/test
diff options
context:
space:
mode:
authorReynold Xin <rxin@databricks.com>2016-08-19 21:11:35 +0800
committerWenchen Fan <wenchen@databricks.com>2016-08-19 21:11:35 +0800
commit67e59d464f782ff5f509234212aa072a7653d7bf (patch)
tree7b7c935a0ead8d5043443a4712584f173ef83342 /sql/catalyst/src/test
parent072acf5e1460d66d4b60b536d5b2ccddeee80794 (diff)
downloadspark-67e59d464f782ff5f509234212aa072a7653d7bf.tar.gz
spark-67e59d464f782ff5f509234212aa072a7653d7bf.tar.bz2
spark-67e59d464f782ff5f509234212aa072a7653d7bf.zip
[SPARK-16994][SQL] Whitelist operators for predicate pushdown
## What changes were proposed in this pull request? This patch changes predicate pushdown optimization rule (PushDownPredicate) from using a blacklist to a whitelist. That is to say, operators must be explicitly allowed. This approach is more future-proof: previously it was possible for us to introduce a new operator and then render the optimization rule incorrect. This also fixes the bug that previously we allowed pushing filter beneath limit, which was incorrect. That is to say, before this patch, the optimizer would rewrite ``` select * from (select * from range(10) limit 5) where id > 3 to select * from range(10) where id > 3 limit 5 ``` ## How was this patch tested? - a unit test case in FilterPushdownSuite - an end-to-end test in limit.sql Author: Reynold Xin <rxin@databricks.com> Closes #14713 from rxin/SPARK-16994.
Diffstat (limited to 'sql/catalyst/src/test')
-rw-r--r--sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala6
1 files changed, 6 insertions, 0 deletions
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
index 596b8fcea1..9f25e9d8e9 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
@@ -111,6 +111,12 @@ class FilterPushdownSuite extends PlanTest {
assert(optimized == correctAnswer)
}
+ test("SPARK-16994: filter should not be pushed through limit") {
+ val originalQuery = testRelation.limit(10).where('a === 1).analyze
+ val optimized = Optimize.execute(originalQuery)
+ comparePlans(optimized, originalQuery)
+ }
+
test("can't push without rewrite") {
val originalQuery =
testRelation