aboutsummaryrefslogtreecommitdiff
path: root/sql/hive/src/main
diff options
context:
space:
mode:
authorLiang-Chi Hsieh <simonh@tw.ibm.com>2016-04-01 13:03:27 -0700
committerMichael Armbrust <michael@databricks.com>2016-04-01 13:08:09 -0700
commita884daad805a701494e87393dc307937472a985d (patch)
treeeba73c6a1a355095924c7c4894498e5c423e698a /sql/hive/src/main
parentdf68beb85de59bb6d35b2a8a3b85dbc447798bf5 (diff)
downloadspark-a884daad805a701494e87393dc307937472a985d.tar.gz
spark-a884daad805a701494e87393dc307937472a985d.tar.bz2
spark-a884daad805a701494e87393dc307937472a985d.zip
[SPARK-14191][SQL] Remove invalid Expand operator constraints
`Expand` operator now uses its child plan's constraints as its valid constraints (i.e., the base of constraints). This is not correct because `Expand` will set its group by attributes to null values. So the nullability of these attributes should be true. E.g., for an `Expand` operator like: val input = LocalRelation('a.int, 'b.int, 'c.int).where('c.attr > 10 && 'a.attr < 5 && 'b.attr > 2) Expand( Seq( Seq('c, Literal.create(null, StringType), 1), Seq('c, 'a, 2)), Seq('c, 'a, 'gid.int), Project(Seq('a, 'c), input)) The `Project` operator has the constraints `IsNotNull('a)`, `IsNotNull('b)` and `IsNotNull('c)`. But the `Expand` should not have `IsNotNull('a)` in its constraints. This PR is the first step for this issue and remove invalid constraints of `Expand` operator. A test is added to `ConstraintPropagationSuite`. Author: Liang-Chi Hsieh <simonh@tw.ibm.com> Author: Michael Armbrust <michael@databricks.com> Closes #11995 from viirya/fix-expand-constraints.
Diffstat (limited to 'sql/hive/src/main')
0 files changed, 0 insertions, 0 deletions