diff options
| author | Sameer Agarwal <sameer@databricks.com> | 2016-03-16 16:26:51 -0700 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2016-03-16 16:26:51 -0700 |
| commit | f96997ba244a14c26e85a2475415a762d0c0d0a8 (patch) | |
| tree | 8278b83a155fc09d736375890a043d9a0f8243d7 /build | |
| parent | b90c0206faeb0883fba1c79fe18aa72affb9988e (diff) | |
| download | spark-f96997ba244a14c26e85a2475415a762d0c0d0a8.tar.gz spark-f96997ba244a14c26e85a2475415a762d0c0d0a8.tar.bz2 spark-f96997ba244a14c26e85a2475415a762d0c0d0a8.zip | |
[SPARK-13871][SQL] Support for inferring filters from data constraints
## What changes were proposed in this pull request?
This PR generalizes the `NullFiltering` optimizer rule in catalyst to `InferFiltersFromConstraints` that can automatically infer all relevant filters based on an operator's constraints while making sure of 2 things:
(a) no redundant filters are generated, and
(b) filters that do not contribute to any further optimizations are not generated.
## How was this patch tested?
Extended all tests in `InferFiltersFromConstraintsSuite` (that were initially based on `NullFilteringSuite` to test filter inference in `Filter` and `Join` operators.
In particular the 2 tests ( `single inner join with pre-existing filters: filter out values on either side` and `multiple inner joins: filter out values on all sides on equi-join keys` attempts to highlight/test the real potential of this rule for join optimization.
Author: Sameer Agarwal <sameer@databricks.com>
Closes #11665 from sameeragarwal/infer-filters.
Diffstat (limited to 'build')
0 files changed, 0 insertions, 0 deletions