diff options
author | Cheng Hao <hao.cheng@intel.com> | 2015-06-22 20:04:49 -0700 |
---|---|---|
committer | Yin Huai <yhuai@databricks.com> | 2015-06-22 20:04:49 -0700 |
commit | 13321e65559f6354ec1287a690580fd6f498ef89 (patch) | |
tree | 82529748f2e083dff5e4e6baad62021543fc37bf /sql/hive/compatibility/src/test/scala | |
parent | 6b7f2ceafdcbb014791909747c2210b527305df9 (diff) | |
download | spark-13321e65559f6354ec1287a690580fd6f498ef89.tar.gz spark-13321e65559f6354ec1287a690580fd6f498ef89.tar.bz2 spark-13321e65559f6354ec1287a690580fd6f498ef89.zip |
[SPARK-7859] [SQL] Collect_set() behavior differences which fails the unit test under jdk8
To reproduce that:
```
JAVA_HOME=/home/hcheng/Java/jdk1.8.0_45 | build/sbt -Phadoop-2.3 -Phive 'test-only org.apache.spark.sql.hive.execution.HiveWindowFunctionQueryWithoutCodeGenSuite'
```
A simple workaround to fix that is update the original query, for getting the output size instead of the exact elements of the array (output by collect_set())
Author: Cheng Hao <hao.cheng@intel.com>
Closes #6402 from chenghao-intel/windowing and squashes the following commits:
99312ad [Cheng Hao] add order by for the select clause
edf8ce3 [Cheng Hao] update the code as suggested
7062da7 [Cheng Hao] fix the collect_set() behaviour differences under different versions of JDK
Diffstat (limited to 'sql/hive/compatibility/src/test/scala')
-rw-r--r-- | sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala b/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala index 934452fe57..31a49a3683 100644 --- a/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala +++ b/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala @@ -526,8 +526,14 @@ abstract class HiveWindowFunctionQueryBaseSuite extends HiveComparisonTest with | rows between 2 preceding and 2 following); """.stripMargin, reset = false) + // collect_set() output array in an arbitrary order, hence causes different result + // when running this test suite under Java 7 and 8. + // We change the original sql query a little bit for making the test suite passed + // under different JDK createQueryTest("windowing.q -- 20. testSTATs", """ + |select p_mfgr,p_name, p_size, sdev, sdev_pop, uniq_data, var, cor, covarp + |from ( |select p_mfgr,p_name, p_size, |stddev(p_retailprice) over w1 as sdev, |stddev_pop(p_retailprice) over w1 as sdev_pop, @@ -538,6 +544,8 @@ abstract class HiveWindowFunctionQueryBaseSuite extends HiveComparisonTest with |from part |window w1 as (distribute by p_mfgr sort by p_mfgr, p_name | rows between 2 preceding and 2 following) + |) t lateral view explode(uniq_size) d as uniq_data + |order by p_mfgr,p_name, p_size, sdev, sdev_pop, uniq_data, var, cor, covarp """.stripMargin, reset = false) createQueryTest("windowing.q -- 21. testDISTs", |