diff options
author | Russell Spitzer <Russell.Spitzer@gmail.com> | 2016-07-11 21:40:09 -0700 |
---|---|---|
committer | Yin Huai <yhuai@databricks.com> | 2016-07-11 21:40:09 -0700 |
commit | b1e5281c5cb429e338c3719c13c0b93078d7312a (patch) | |
tree | 044424e0f5fb922f2136d8b88f221a38ccc6c9f1 /.gitignore | |
parent | 9cc74f95edb6e4f56151966139cd0dc24e377949 (diff) | |
download | spark-b1e5281c5cb429e338c3719c13c0b93078d7312a.tar.gz spark-b1e5281c5cb429e338c3719c13c0b93078d7312a.tar.bz2 spark-b1e5281c5cb429e338c3719c13c0b93078d7312a.zip |
[SPARK-12639][SQL] Mark Filters Fully Handled By Sources with *
## What changes were proposed in this pull request?
In order to make it clear which filters are fully handled by the
underlying datasource we will mark them with an *. This will give a
clear visual queue to users that the filter is being treated differently
by catalyst than filters which are just presented to the underlying
DataSource.
Examples from the FilteredScanSuite, in this example `c IN (...)` is handled by the source, `b < ...` is not
### Before
```
//SELECT a FROM oneToTenFiltered WHERE a + b > 9 AND b < 16 AND c IN ('bbbbbBBBBB', 'cccccCCCCC', 'dddddDDDDD', 'foo')
== Physical Plan ==
Project [a#0]
+- Filter (((a#0 + b#1) > 9) && (b#1 < 16))
+- Scan SimpleFilteredScan(1,10)[a#0,b#1] PushedFilters: [LessThan(b,16), In(c, [bbbbbBBBBB,cccccCCCCC,dddddDDDDD,foo]]
```
### After
```
== Physical Plan ==
Project [a#0]
+- Filter (((a#0 + b#1) > 9) && (b#1 < 16))
+- Scan SimpleFilteredScan(1,10)[a#0,b#1] PushedFilters: [LessThan(b,16), *In(c, [bbbbbBBBBB,cccccCCCCC,dddddDDDDD,foo]]
```
## How was the this patch tested?
Manually tested with the Spark Cassandra Connector, a source which fully handles underlying filters. Now fully handled filters appear with an * next to their names. I can add an automated test as well if requested
Post 1.6.1
Tested by modifying the FilteredScanSuite to run explains.
Author: Russell Spitzer <Russell.Spitzer@gmail.com>
Closes #11317 from RussellSpitzer/SPARK-12639-Star.
Diffstat (limited to '.gitignore')
0 files changed, 0 insertions, 0 deletions