diff options
author | gatorsmile <gatorsmile@gmail.com> | 2016-11-30 19:40:58 +0800 |
---|---|---|
committer | Wenchen Fan <wenchen@databricks.com> | 2016-11-30 19:40:58 +0800 |
commit | 2eb093decb5e87a1ea71bbaa28092876a8c84996 (patch) | |
tree | 00251a85adfdafa9c927d9c74f7cdf57f6e05179 /.gitattributes | |
parent | c5a64d760600ff430899e401751c41dc6b27cee6 (diff) | |
download | spark-2eb093decb5e87a1ea71bbaa28092876a8c84996.tar.gz spark-2eb093decb5e87a1ea71bbaa28092876a8c84996.tar.bz2 spark-2eb093decb5e87a1ea71bbaa28092876a8c84996.zip |
[SPARK-17897][SQL] Fixed IsNotNull Constraint Inference Rule
### What changes were proposed in this pull request?
The `constraints` of an operator is the expressions that evaluate to `true` for all the rows produced. That means, the expression result should be neither `false` nor `unknown` (NULL). Thus, we can conclude that `IsNotNull` on all the constraints, which are generated by its own predicates or propagated from the children. The constraint can be a complex expression. For better usage of these constraints, we try to push down `IsNotNull` to the lowest-level expressions (i.e., `Attribute`). `IsNotNull` can be pushed through an expression when it is null intolerant. (When the input is NULL, the null-intolerant expression always evaluates to NULL.)
Below is the existing code we have for `IsNotNull` pushdown.
```Scala
private def scanNullIntolerantExpr(expr: Expression): Seq[Attribute] = expr match {
case a: Attribute => Seq(a)
case _: NullIntolerant | IsNotNull(_: NullIntolerant) =>
expr.children.flatMap(scanNullIntolerantExpr)
case _ => Seq.empty[Attribute]
}
```
**`IsNotNull` itself is not null-intolerant.** It converts `null` to `false`. If the expression does not include any `Not`-like expression, it works; otherwise, it could generate a wrong result. This PR is to fix the above function by removing the `IsNotNull` from the inference. After the fix, when a constraint has a `IsNotNull` expression, we infer new attribute-specific `IsNotNull` constraints if and only if `IsNotNull` appears in the root.
Without the fix, the following test case will return empty.
```Scala
val data = Seq[java.lang.Integer](1, null).toDF("key")
data.filter("not key is not null").show()
```
Before the fix, the optimized plan is like
```
== Optimized Logical Plan ==
Project [value#1 AS key#3]
+- Filter (isnotnull(value#1) && NOT isnotnull(value#1))
+- LocalRelation [value#1]
```
After the fix, the optimized plan is like
```
== Optimized Logical Plan ==
Project [value#1 AS key#3]
+- Filter NOT isnotnull(value#1)
+- LocalRelation [value#1]
```
### How was this patch tested?
Added a test
Author: gatorsmile <gatorsmile@gmail.com>
Closes #16067 from gatorsmile/isNotNull2.
Diffstat (limited to '.gitattributes')
0 files changed, 0 insertions, 0 deletions