diff options
author | Kousuke Saruta <sarutak@oss.nttdata.co.jp> | 2014-10-08 17:03:47 -0700 |
---|---|---|
committer | Michael Armbrust <michael@databricks.com> | 2014-10-08 17:03:47 -0700 |
commit | a85f24accd3266e0f97ee04d03c22b593d99c062 (patch) | |
tree | 13c80dabf7f3e5e79120b089be2de685916cbd1b /python/pyspark/sql.py | |
parent | add174aa56d291bc48ef73a42c39428c923efe31 (diff) | |
download | spark-a85f24accd3266e0f97ee04d03c22b593d99c062.tar.gz spark-a85f24accd3266e0f97ee04d03c22b593d99c062.tar.bz2 spark-a85f24accd3266e0f97ee04d03c22b593d99c062.zip |
[SPARK-3831] [SQL] Filter rule Improvement and bool expression optimization.
If we write the filter which is always FALSE like
SELECT * from person WHERE FALSE;
200 tasks will run. I think, 1 task is enough.
And current optimizer cannot optimize the case NOT is duplicated like
SELECT * from person WHERE NOT ( NOT (age > 30));
The filter rule above should be simplified
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Closes #2692 from sarutak/SPARK-3831 and squashes the following commits:
25f3e20 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3831
23c750c [Kousuke Saruta] Improved unsupported predicate test case
a11b9f3 [Kousuke Saruta] Modified NOT predicate test case in PartitionBatchPruningSuite
8ea872b [Kousuke Saruta] Fixed the number of tasks when the data of LocalRelation is empty.
Diffstat (limited to 'python/pyspark/sql.py')
0 files changed, 0 insertions, 0 deletions