diff options
| author | gatorsmile <gatorsmile@gmail.com> | 2016-02-01 11:22:02 -0800 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2016-02-01 11:22:02 -0800 |
| commit | 33c8a490f7f64320c53530a57bd8d34916e3607c (patch) | |
| tree | dba1fa3dc9b1b12af869425ac6dd49174ac64d37 /dev/README.md | |
| parent | 6075573a93176ee8c071888e4525043d9e73b061 (diff) | |
| download | spark-33c8a490f7f64320c53530a57bd8d34916e3607c.tar.gz spark-33c8a490f7f64320c53530a57bd8d34916e3607c.tar.bz2 spark-33c8a490f7f64320c53530a57bd8d34916e3607c.zip | |
[SPARK-12989][SQL] Delaying Alias Cleanup after ExtractWindowExpressions
JIRA: https://issues.apache.org/jira/browse/SPARK-12989
In the rule `ExtractWindowExpressions`, we simply replace alias by the corresponding attribute. However, this will cause an issue exposed by the following case:
```scala
val data = Seq(("a", "b", "c", 3), ("c", "b", "a", 3)).toDF("A", "B", "C", "num")
.withColumn("Data", struct("A", "B", "C"))
.drop("A")
.drop("B")
.drop("C")
val winSpec = Window.partitionBy("Data.A", "Data.B").orderBy($"num".desc)
data.select($"*", max("num").over(winSpec) as "max").explain(true)
```
In this case, both `Data.A` and `Data.B` are `alias` in `WindowSpecDefinition`. If we replace these alias expression by their alias names, we are unable to know what they are since they will not be put in `missingExpr` too.
Author: gatorsmile <gatorsmile@gmail.com>
Author: xiaoli <lixiao1983@gmail.com>
Author: Xiao Li <xiaoli@Xiaos-MacBook-Pro.local>
Closes #10963 from gatorsmile/seletStarAfterColDrop.
Diffstat (limited to 'dev/README.md')
0 files changed, 0 insertions, 0 deletions