diff options
author | Liang-Chi Hsieh <viirya@appier.com> | 2015-12-01 21:51:33 -0800 |
---|---|---|
committer | Yin Huai <yhuai@databricks.com> | 2015-12-01 21:51:33 -0800 |
commit | 0f37d1d7ed7f6e34f98f2a3c274918de29e7a1d7 (patch) | |
tree | f21a642dbf33441ef71aaaafd47af0d30d911057 | |
parent | 8a75a3049539eeef04c0db51736e97070c162b46 (diff) | |
download | spark-0f37d1d7ed7f6e34f98f2a3c274918de29e7a1d7.tar.gz spark-0f37d1d7ed7f6e34f98f2a3c274918de29e7a1d7.tar.bz2 spark-0f37d1d7ed7f6e34f98f2a3c274918de29e7a1d7.zip |
[SPARK-11949][SQL] Check bitmasks to set nullable property
Following up #10038.
We can use bitmasks to determine which grouping expressions need to be set as nullable.
cc yhuai
Author: Liang-Chi Hsieh <viirya@appier.com>
Closes #10067 from viirya/fix-cube-following.
-rw-r--r-- | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala | 13 |
1 files changed, 9 insertions, 4 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala index 765327c474..d3163dcd4d 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala @@ -224,10 +224,15 @@ class Analyzer( case other => Alias(other, other.toString)() } - // TODO: We need to use bitmasks to determine which grouping expressions need to be - // set as nullable. For example, if we have GROUPING SETS ((a,b), a), we do not need - // to change the nullability of a. - val attributeMap = groupByAliases.map(a => (a -> a.toAttribute.withNullability(true))).toMap + val nonNullBitmask = x.bitmasks.reduce(_ & _) + + val attributeMap = groupByAliases.zipWithIndex.map { case (a, idx) => + if ((nonNullBitmask & 1 << idx) == 0) { + (a -> a.toAttribute.withNullability(true)) + } else { + (a -> a.toAttribute) + } + }.toMap val aggregations: Seq[NamedExpression] = x.aggregations.map { // If an expression is an aggregate (contains a AggregateExpression) then we dont change |