diff options
author | Takuya UESHIN <ueshin@happy-camper.st> | 2014-08-11 20:15:01 -0700 |
---|---|---|
committer | Michael Armbrust <michael@databricks.com> | 2014-08-11 20:15:01 -0700 |
commit | c9c89c31b6114832fe282c21fecd663d8105b9bc (patch) | |
tree | 25101eaa54f9abbe8ea95dc075c4b36a5ebc2da3 | |
parent | 647aeba3a9e101d35083f7c4afbcfe7a33f7fc62 (diff) | |
download | spark-c9c89c31b6114832fe282c21fecd663d8105b9bc.tar.gz spark-c9c89c31b6114832fe282c21fecd663d8105b9bc.tar.bz2 spark-c9c89c31b6114832fe282c21fecd663d8105b9bc.zip |
[SPARK-2965][SQL] Fix HashOuterJoin output nullabilities.
Output attributes of opposite side of `OuterJoin` should be nullable.
Author: Takuya UESHIN <ueshin@happy-camper.st>
Closes #1887 from ueshin/issues/SPARK-2965 and squashes the following commits:
bcb2d37 [Takuya UESHIN] Fix HashOuterJoin output nullabilities.
-rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala | 13 |
1 files changed, 12 insertions, 1 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala index 51bb615307..ea075f8c65 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala @@ -168,7 +168,18 @@ case class HashOuterJoin( override def requiredChildDistribution = ClusteredDistribution(leftKeys) :: ClusteredDistribution(rightKeys) :: Nil - def output = left.output ++ right.output + override def output = { + joinType match { + case LeftOuter => + left.output ++ right.output.map(_.withNullability(true)) + case RightOuter => + left.output.map(_.withNullability(true)) ++ right.output + case FullOuter => + left.output.map(_.withNullability(true)) ++ right.output.map(_.withNullability(true)) + case x => + throw new Exception(s"HashOuterJoin should not take $x as the JoinType") + } + } // TODO we need to rewrite all of the iterators with our own implementation instead of the Scala // iterator for performance purpose. |