diff options
| author | Herman van Hovell <hvanhovell@databricks.com> | 2016-11-15 06:59:25 -0800 |
|---|---|---|
| committer | gatorsmile <gatorsmile@gmail.com> | 2016-11-15 06:59:25 -0800 |
| commit | f14ae4900ad0ed66ba36108b7792d56cd6767a69 (patch) | |
| tree | cb078120aadb305c7c2f6d5210eafb199aca06a3 /sql/core/src/test/resources/sql-tests/results | |
| parent | 33be4da5391b884191c405ffbce7d382ea8a2f66 (diff) | |
| download | spark-f14ae4900ad0ed66ba36108b7792d56cd6767a69.tar.gz spark-f14ae4900ad0ed66ba36108b7792d56cd6767a69.tar.bz2 spark-f14ae4900ad0ed66ba36108b7792d56cd6767a69.zip | |
[SPARK-18300][SQL] Do not apply foldable propagation with expand as a child.
## What changes were proposed in this pull request?
The `FoldablePropagation` optimizer rule, pulls foldable values out from under an `Expand`. This breaks the `Expand` in two ways:
- It rewrites the output attributes of the `Expand`. We explicitly define output attributes for `Expand`, these are (unfortunately) considered as part of the expressions of the `Expand` and can be rewritten.
- Expand can actually change the column (it will typically re-use the attributes or the underlying plan). This means that we cannot safely propagate the expressions from under an `Expand`.
This PR fixes this and (hopefully) other issues by explicitly whitelisting allowed operators.
## How was this patch tested?
Added tests to `FoldablePropagationSuite` and to `SQLQueryTestSuite`.
Author: Herman van Hovell <hvanhovell@databricks.com>
Closes #15857 from hvanhovell/SPARK-18300.
Diffstat (limited to 'sql/core/src/test/resources/sql-tests/results')
| -rw-r--r-- | sql/core/src/test/resources/sql-tests/results/group-by.sql.out | 10 |
1 files changed, 9 insertions, 1 deletions
diff --git a/sql/core/src/test/resources/sql-tests/results/group-by.sql.out b/sql/core/src/test/resources/sql-tests/results/group-by.sql.out index af6c930d64..4b87d5161f 100644 --- a/sql/core/src/test/resources/sql-tests/results/group-by.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/group-by.sql.out @@ -1,5 +1,5 @@ -- Automatically generated by SQLQueryTestSuite --- Number of queries: 14 +-- Number of queries: 15 -- !query 0 @@ -131,3 +131,11 @@ FROM testData struct<skewness(CAST(a AS DOUBLE)):double,kurtosis(CAST(a AS DOUBLE)):double,min(a):int,max(a):int,avg(a):double,var_samp(CAST(a AS DOUBLE)):double,stddev_samp(CAST(a AS DOUBLE)):double,sum(a):bigint,count(a):bigint> -- !query 13 output -0.2723801058145729 -1.5069204152249134 1 3 2.142857142857143 0.8095238095238094 0.8997354108424372 15 7 + + +-- !query 14 +SELECT COUNT(DISTINCT b), COUNT(DISTINCT b, c) FROM (SELECT 1 AS a, 2 AS b, 3 AS c) GROUP BY a +-- !query 14 schema +struct<count(DISTINCT b):bigint,count(DISTINCT b, c):bigint> +-- !query 14 output +1 1 |