diff options
author | Yijie Shen <henry.yijieshen@gmail.com> | 2015-09-18 13:20:13 -0700 |
---|---|---|
committer | Yin Huai <yhuai@databricks.com> | 2015-09-18 13:20:13 -0700 |
commit | c6f8135ee52202bd86adb090ab631e80330ea4df (patch) | |
tree | 3e29ff33c4d57bf44595961b8f28bdb411544919 /sql/core | |
parent | 00a2911c5bea67a1a4796fb1d6fd5d0a8ee79001 (diff) | |
download | spark-c6f8135ee52202bd86adb090ab631e80330ea4df.tar.gz spark-c6f8135ee52202bd86adb090ab631e80330ea4df.tar.bz2 spark-c6f8135ee52202bd86adb090ab631e80330ea4df.zip |
[SPARK-10539] [SQL] Project should not be pushed down through Intersect or Except #8742
Intersect and Except are both set operators and they use the all the columns to compare equality between rows. When pushing their Project parent down, the relations they based on would change, therefore not an equivalent transformation.
JIRA: https://issues.apache.org/jira/browse/SPARK-10539
I added some comments based on the fix of https://github.com/apache/spark/pull/8742.
Author: Yijie Shen <henry.yijieshen@gmail.com>
Author: Yin Huai <yhuai@databricks.com>
Closes #8823 from yhuai/fix_set_optimization.
Diffstat (limited to 'sql/core')
-rw-r--r-- | sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala | 9 |
1 files changed, 9 insertions, 0 deletions
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala index c167999af5..1370713975 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala @@ -907,4 +907,13 @@ class DataFrameSuite extends QueryTest with SharedSQLContext { assert(row.getDouble(1) - row.getDouble(3) === 0.0 +- 0.001) } } + + test("SPARK-10539: Project should not be pushed down through Intersect or Except") { + val df1 = (1 to 100).map(Tuple1.apply).toDF("i") + val df2 = (1 to 30).map(Tuple1.apply).toDF("i") + val intersect = df1.intersect(df2) + val except = df1.except(df2) + assert(intersect.count() === 30) + assert(except.count() === 70) + } } |