aboutsummaryrefslogtreecommitdiff
path: root/.gitattributes
diff options
context:
space:
mode:
authorgatorsmile <gatorsmile@gmail.com>2016-11-30 19:40:58 +0800
committerWenchen Fan <wenchen@databricks.com>2016-11-30 19:40:58 +0800
commit2eb093decb5e87a1ea71bbaa28092876a8c84996 (patch)
tree00251a85adfdafa9c927d9c74f7cdf57f6e05179 /.gitattributes
parentc5a64d760600ff430899e401751c41dc6b27cee6 (diff)
downloadspark-2eb093decb5e87a1ea71bbaa28092876a8c84996.tar.gz
spark-2eb093decb5e87a1ea71bbaa28092876a8c84996.tar.bz2
spark-2eb093decb5e87a1ea71bbaa28092876a8c84996.zip
[SPARK-17897][SQL] Fixed IsNotNull Constraint Inference Rule
### What changes were proposed in this pull request? The `constraints` of an operator is the expressions that evaluate to `true` for all the rows produced. That means, the expression result should be neither `false` nor `unknown` (NULL). Thus, we can conclude that `IsNotNull` on all the constraints, which are generated by its own predicates or propagated from the children. The constraint can be a complex expression. For better usage of these constraints, we try to push down `IsNotNull` to the lowest-level expressions (i.e., `Attribute`). `IsNotNull` can be pushed through an expression when it is null intolerant. (When the input is NULL, the null-intolerant expression always evaluates to NULL.) Below is the existing code we have for `IsNotNull` pushdown. ```Scala private def scanNullIntolerantExpr(expr: Expression): Seq[Attribute] = expr match { case a: Attribute => Seq(a) case _: NullIntolerant | IsNotNull(_: NullIntolerant) => expr.children.flatMap(scanNullIntolerantExpr) case _ => Seq.empty[Attribute] } ``` **`IsNotNull` itself is not null-intolerant.** It converts `null` to `false`. If the expression does not include any `Not`-like expression, it works; otherwise, it could generate a wrong result. This PR is to fix the above function by removing the `IsNotNull` from the inference. After the fix, when a constraint has a `IsNotNull` expression, we infer new attribute-specific `IsNotNull` constraints if and only if `IsNotNull` appears in the root. Without the fix, the following test case will return empty. ```Scala val data = Seq[java.lang.Integer](1, null).toDF("key") data.filter("not key is not null").show() ``` Before the fix, the optimized plan is like ``` == Optimized Logical Plan == Project [value#1 AS key#3] +- Filter (isnotnull(value#1) && NOT isnotnull(value#1)) +- LocalRelation [value#1] ``` After the fix, the optimized plan is like ``` == Optimized Logical Plan == Project [value#1 AS key#3] +- Filter NOT isnotnull(value#1) +- LocalRelation [value#1] ``` ### How was this patch tested? Added a test Author: gatorsmile <gatorsmile@gmail.com> Closes #16067 from gatorsmile/isNotNull2.
Diffstat (limited to '.gitattributes')
0 files changed, 0 insertions, 0 deletions