diff options
| author | gatorsmile <gatorsmile@gmail.com> | 2016-03-09 12:50:55 -0800 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2016-03-09 12:50:55 -0800 |
| commit | c6aa356cd831ea2d159568b699bd5b791f3d8f25 (patch) | |
| tree | 1d375e4cdf7aebe9b18ffed860ee9f1b8c2e0d3b /mllib | |
| parent | 3dc9ae2e158e5b51df6f799767946fe1d190156b (diff) | |
| download | spark-c6aa356cd831ea2d159568b699bd5b791f3d8f25.tar.gz spark-c6aa356cd831ea2d159568b699bd5b791f3d8f25.tar.bz2 spark-c6aa356cd831ea2d159568b699bd5b791f3d8f25.zip | |
[SPARK-13527][SQL] Prune Filters based on Constraints
#### What changes were proposed in this pull request?
Remove all the deterministic conditions in a [[Filter]] that are contained in the Child's Constraints.
For example, the first query can be simplified to the second one.
```scala
val queryWithUselessFilter = tr1
.where("tr1.a".attr > 10 || "tr1.c".attr < 10)
.join(tr2.where('d.attr < 100), Inner, Some("tr1.a".attr === "tr2.a".attr))
.where(
("tr1.a".attr > 10 || "tr1.c".attr < 10) &&
'd.attr < 100 &&
"tr2.a".attr === "tr1.a".attr)
```
```scala
val query = tr1
.where("tr1.a".attr > 10 || "tr1.c".attr < 10)
.join(tr2.where('d.attr < 100), Inner, Some("tr1.a".attr === "tr2.a".attr))
```
#### How was this patch tested?
Six test cases are added.
Author: gatorsmile <gatorsmile@gmail.com>
Closes #11406 from gatorsmile/FilterRemoval.
Diffstat (limited to 'mllib')
0 files changed, 0 insertions, 0 deletions