diff options
| author | eyal farago <eyal farago> | 2016-11-01 17:12:20 +0100 |
|---|---|---|
| committer | Herman van Hovell <hvanhovell@databricks.com> | 2016-11-01 17:12:20 +0100 |
| commit | 5441a6269e00e3903ae6c1ea8deb4ddf3d2e9975 (patch) | |
| tree | 54c00373573510b26265761cdb86846c01a704a2 /sql/core/src/test | |
| parent | f7c145d8ce14b23019099c509d5a2b6dfb1fe62c (diff) | |
| download | spark-5441a6269e00e3903ae6c1ea8deb4ddf3d2e9975.tar.gz spark-5441a6269e00e3903ae6c1ea8deb4ddf3d2e9975.tar.bz2 spark-5441a6269e00e3903ae6c1ea8deb4ddf3d2e9975.zip | |
[SPARK-16839][SQL] redundant aliases after cleanupAliases
## What changes were proposed in this pull request?
Simplify struct creation, especially the aspect of `CleanupAliases` which missed some aliases when handling trees created by `CreateStruct`.
This PR includes:
1. A failing test (create struct with nested aliases, some of the aliases survive `CleanupAliases`).
2. A fix that transforms `CreateStruct` into a `CreateNamedStruct` constructor, effectively eliminating `CreateStruct` from all expression trees.
3. A `NamePlaceHolder` used by `CreateStruct` when column names cannot be extracted from unresolved `NamedExpression`.
4. A new Analyzer rule that resolves `NamePlaceHolder` into a string literal once the `NamedExpression` is resolved.
5. `CleanupAliases` code was simplified as it no longer has to deal with `CreateStruct`'s top level columns.
## How was this patch tested?
running all tests-suits in package org.apache.spark.sql, especially including the analysis suite, making sure added test initially fails, after applying suggested fix rerun the entire analysis package successfully.
modified few tests that expected `CreateStruct` which is now transformed into `CreateNamedStruct`.
Credit goes to hvanhovell for assisting with this PR.
Author: eyal farago <eyal farago>
Author: eyal farago <eyal.farago@gmail.com>
Author: Herman van Hovell <hvanhovell@databricks.com>
Author: Eyal Farago <eyal.farago@actimize.com>
Author: Hyukjin Kwon <gurwls223@gmail.com>
Author: eyalfa <eyal.farago@gmail.com>
Closes #14444 from eyalfa/SPARK-16839_redundant_aliases_after_cleanupAliases.
Diffstat (limited to 'sql/core/src/test')
| -rw-r--r-- | sql/core/src/test/resources/sql-tests/inputs/group-by.sql | 2 | ||||
| -rw-r--r-- | sql/core/src/test/resources/sql-tests/results/group-by.sql.out | 4 |
2 files changed, 3 insertions, 3 deletions
diff --git a/sql/core/src/test/resources/sql-tests/inputs/group-by.sql b/sql/core/src/test/resources/sql-tests/inputs/group-by.sql index 6741703d9d..d496af686d 100644 --- a/sql/core/src/test/resources/sql-tests/inputs/group-by.sql +++ b/sql/core/src/test/resources/sql-tests/inputs/group-by.sql @@ -14,4 +14,4 @@ select 'foo' from myview where int_col == 0 group by 1; select 'foo', approx_count_distinct(int_col) from myview where int_col == 0 group by 1; -- group-by should not produce any rows (sort aggregate). -select 'foo', max(struct(int_col)) from myview where int_col == 0 group by 1; +select 'foo', max(struct(int_col)) as agg_struct from myview where int_col == 0 group by 1; diff --git a/sql/core/src/test/resources/sql-tests/results/group-by.sql.out b/sql/core/src/test/resources/sql-tests/results/group-by.sql.out index 9127bd4dd4..dede3a09ce 100644 --- a/sql/core/src/test/resources/sql-tests/results/group-by.sql.out +++ b/sql/core/src/test/resources/sql-tests/results/group-by.sql.out @@ -44,8 +44,8 @@ struct<foo:string,approx_count_distinct(int_col):bigint> -- !query 5 -select 'foo', max(struct(int_col)) from myview where int_col == 0 group by 1 +select 'foo', max(struct(int_col)) as agg_struct from myview where int_col == 0 group by 1 -- !query 5 schema -struct<foo:string,max(struct(int_col)):struct<int_col:int>> +struct<foo:string,agg_struct:struct<int_col:int>> -- !query 5 output |