diff options
author | wangyang <wangyang@haizhi.com> | 2016-11-05 14:32:28 +0100 |
---|---|---|
committer | Herman van Hovell <hvanhovell@databricks.com> | 2016-11-05 14:32:28 +0100 |
commit | fb0d60814a79747beb68da9613679141c44f2540 (patch) | |
tree | 8bf2e4e959b546a9a69bd2b3ab0841869ab9d9ce /sql/core/src/test | |
parent | a87471c83006ec11c372b4f915e17a0501f1f536 (diff) | |
download | spark-fb0d60814a79747beb68da9613679141c44f2540.tar.gz spark-fb0d60814a79747beb68da9613679141c44f2540.tar.bz2 spark-fb0d60814a79747beb68da9613679141c44f2540.zip |
[SPARK-17849][SQL] Fix NPE problem when using grouping sets
## What changes were proposed in this pull request?
Prior this pr, the following code would cause an NPE:
`case class point(a:String, b:String, c:String, d: Int)`
`val data = Seq(
point("1","2","3", 1),
point("4","5","6", 1),
point("7","8","9", 1)
)`
`sc.parallelize(data).toDF().registerTempTable("table")`
`spark.sql("select a, b, c, count(d) from table group by a, b, c GROUPING SETS ((a)) ").show()`
The reason is that when the grouping_id() behavior was changed in #10677, some code (which should be changed) was left out.
Take the above code for example, prior #10677, the bit mask for set "(a)" was `001`, while after #10677 the bit mask was changed to `011`. However, the `nonNullBitmask` was not changed accordingly.
This pr will fix this problem.
## How was this patch tested?
add integration tests
Author: wangyang <wangyang@haizhi.com>
Closes #15416 from yangw1234/groupingid.
Diffstat (limited to 'sql/core/src/test')
-rw-r--r-- | sql/core/src/test/resources/sql-tests/inputs/grouping_set.sql | 17 | ||||
-rw-r--r-- | sql/core/src/test/resources/sql-tests/results/grouping_set.sql.out | 42 |
2 files changed, 59 insertions, 0 deletions
diff --git a/sql/core/src/test/resources/sql-tests/inputs/grouping_set.sql b/sql/core/src/test/resources/sql-tests/inputs/grouping_set.sql new file mode 100644 index 0000000000..3594283505 --- /dev/null +++ b/sql/core/src/test/resources/sql-tests/inputs/grouping_set.sql @@ -0,0 +1,17 @@ +CREATE TEMPORARY VIEW grouping AS SELECT * FROM VALUES + ("1", "2", "3", 1), + ("4", "5", "6", 1), + ("7", "8", "9", 1) + as grouping(a, b, c, d); + +-- SPARK-17849: grouping set throws NPE #1 +SELECT a, b, c, count(d) FROM grouping GROUP BY a, b, c GROUPING SETS (()); + +-- SPARK-17849: grouping set throws NPE #2 +SELECT a, b, c, count(d) FROM grouping GROUP BY a, b, c GROUPING SETS ((a)); + +-- SPARK-17849: grouping set throws NPE #3 +SELECT a, b, c, count(d) FROM grouping GROUP BY a, b, c GROUPING SETS ((c)); + + + diff --git a/sql/core/src/test/resources/sql-tests/results/grouping_set.sql.out b/sql/core/src/test/resources/sql-tests/results/grouping_set.sql.out new file mode 100644 index 0000000000..edb38a52b7 --- /dev/null +++ b/sql/core/src/test/resources/sql-tests/results/grouping_set.sql.out @@ -0,0 +1,42 @@ +-- Automatically generated by SQLQueryTestSuite +-- Number of queries: 4 + + +-- !query 0 +CREATE TEMPORARY VIEW grouping AS SELECT * FROM VALUES + ("1", "2", "3", 1), + ("4", "5", "6", 1), + ("7", "8", "9", 1) + as grouping(a, b, c, d) +-- !query 0 schema +struct<> +-- !query 0 output + + + +-- !query 1 +SELECT a, b, c, count(d) FROM grouping GROUP BY a, b, c GROUPING SETS (()) +-- !query 1 schema +struct<a:string,b:string,c:string,count(d):bigint> +-- !query 1 output +NULL NULL NULL 3 + + +-- !query 2 +SELECT a, b, c, count(d) FROM grouping GROUP BY a, b, c GROUPING SETS ((a)) +-- !query 2 schema +struct<a:string,b:string,c:string,count(d):bigint> +-- !query 2 output +1 NULL NULL 1 +4 NULL NULL 1 +7 NULL NULL 1 + + +-- !query 3 +SELECT a, b, c, count(d) FROM grouping GROUP BY a, b, c GROUPING SETS ((c)) +-- !query 3 schema +struct<a:string,b:string,c:string,count(d):bigint> +-- !query 3 output +NULL NULL 3 1 +NULL NULL 6 1 +NULL NULL 9 1 |