diff options
author | Takeshi YAMAMURO <linguin.m.s@gmail.com> | 2016-08-25 12:39:58 +0200 |
---|---|---|
committer | Herman van Hovell <hvanhovell@databricks.com> | 2016-08-25 12:39:58 +0200 |
commit | 2b0cc4e0dfa4ffb9f21ff4a303015bc9c962d42b (patch) | |
tree | fcf6b3fea6c9604bee57830c007f9d82ea6937c3 /sql/catalyst | |
parent | 6b8cb1fe52e2c8b4b87b0c7d820f3a1824287328 (diff) | |
download | spark-2b0cc4e0dfa4ffb9f21ff4a303015bc9c962d42b.tar.gz spark-2b0cc4e0dfa4ffb9f21ff4a303015bc9c962d42b.tar.bz2 spark-2b0cc4e0dfa4ffb9f21ff4a303015bc9c962d42b.zip |
[SPARK-12978][SQL] Skip unnecessary final group-by when input data already clustered with group-by keys
This ticket targets the optimization to skip an unnecessary group-by operation below;
Without opt.:
```
== Physical Plan ==
TungstenAggregate(key=[col0#159], functions=[(sum(col1#160),mode=Final,isDistinct=false),(avg(col2#161),mode=Final,isDistinct=false)], output=[col0#159,sum(col1)#177,avg(col2)#178])
+- TungstenAggregate(key=[col0#159], functions=[(sum(col1#160),mode=Partial,isDistinct=false),(avg(col2#161),mode=Partial,isDistinct=false)], output=[col0#159,sum#200,sum#201,count#202L])
+- TungstenExchange hashpartitioning(col0#159,200), None
+- InMemoryColumnarTableScan [col0#159,col1#160,col2#161], InMemoryRelation [col0#159,col1#160,col2#161], true, 10000, StorageLevel(true, true, false, true, 1), ConvertToUnsafe, None
```
With opt.:
```
== Physical Plan ==
TungstenAggregate(key=[col0#159], functions=[(sum(col1#160),mode=Complete,isDistinct=false),(avg(col2#161),mode=Final,isDistinct=false)], output=[col0#159,sum(col1)#177,avg(col2)#178])
+- TungstenExchange hashpartitioning(col0#159,200), None
+- InMemoryColumnarTableScan [col0#159,col1#160,col2#161], InMemoryRelation [col0#159,col1#160,col2#161], true, 10000, StorageLevel(true, true, false, true, 1), ConvertToUnsafe, None
```
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Closes #10896 from maropu/SkipGroupbySpike.
Diffstat (limited to 'sql/catalyst')
0 files changed, 0 insertions, 0 deletions