diff options
| author | Cheng Lian <lian@databricks.com> | 2015-12-28 08:48:44 -0800 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2015-12-28 08:48:44 -0800 |
| commit | 8e23d8db7f28a97e2f4394cdf9d4c4260abbd750 (patch) | |
| tree | c34d6ddda07daa81ab2e37a59333ff92cf39a75f /dev | |
| parent | 8d4940092141c0909b673607f393cdd53f093ed6 (diff) | |
| download | spark-8e23d8db7f28a97e2f4394cdf9d4c4260abbd750.tar.gz spark-8e23d8db7f28a97e2f4394cdf9d4c4260abbd750.tar.bz2 spark-8e23d8db7f28a97e2f4394cdf9d4c4260abbd750.zip | |
[SPARK-12218] Fixes ORC conjunction predicate push down
This PR is a follow-up of PR #10362.
Two major changes:
1. The fix introduced in #10362 is OK for Parquet, but may disable ORC PPD in many cases
PR #10362 stops converting an `AND` predicate if any branch is inconvertible. On the other hand, `OrcFilters` combines all filters into a single big conjunction first and then tries to convert it into ORC `SearchArgument`. This means, if any filter is inconvertible, no filters can be pushed down. This PR fixes this issue by finding out all convertible filters first before doing the actual conversion.
The reason behind the current implementation is mostly due to the limitation of ORC `SearchArgument` builder, which is documented in this PR in detail.
1. Copied the `AND` predicate fix for ORC from #10362 to avoid merge conflict.
Same as #10362, this PR targets master (2.0.0-SNAPSHOT), branch-1.6, and branch-1.5.
Author: Cheng Lian <lian@databricks.com>
Closes #10377 from liancheng/spark-12218.fix-orc-conjunction-ppd.
Diffstat (limited to 'dev')
0 files changed, 0 insertions, 0 deletions