aboutsummaryrefslogtreecommitdiff
path: root/sql/hive-thriftserver
diff options
context:
space:
mode:
authorTakeshi YAMAMURO <linguin.m.s@gmail.com>2016-08-25 12:39:58 +0200
committerHerman van Hovell <hvanhovell@databricks.com>2016-08-25 12:39:58 +0200
commit2b0cc4e0dfa4ffb9f21ff4a303015bc9c962d42b (patch)
treefcf6b3fea6c9604bee57830c007f9d82ea6937c3 /sql/hive-thriftserver
parent6b8cb1fe52e2c8b4b87b0c7d820f3a1824287328 (diff)
downloadspark-2b0cc4e0dfa4ffb9f21ff4a303015bc9c962d42b.tar.gz
spark-2b0cc4e0dfa4ffb9f21ff4a303015bc9c962d42b.tar.bz2
spark-2b0cc4e0dfa4ffb9f21ff4a303015bc9c962d42b.zip
[SPARK-12978][SQL] Skip unnecessary final group-by when input data already clustered with group-by keys
This ticket targets the optimization to skip an unnecessary group-by operation below; Without opt.: ``` == Physical Plan == TungstenAggregate(key=[col0#159], functions=[(sum(col1#160),mode=Final,isDistinct=false),(avg(col2#161),mode=Final,isDistinct=false)], output=[col0#159,sum(col1)#177,avg(col2)#178]) +- TungstenAggregate(key=[col0#159], functions=[(sum(col1#160),mode=Partial,isDistinct=false),(avg(col2#161),mode=Partial,isDistinct=false)], output=[col0#159,sum#200,sum#201,count#202L]) +- TungstenExchange hashpartitioning(col0#159,200), None +- InMemoryColumnarTableScan [col0#159,col1#160,col2#161], InMemoryRelation [col0#159,col1#160,col2#161], true, 10000, StorageLevel(true, true, false, true, 1), ConvertToUnsafe, None ``` With opt.: ``` == Physical Plan == TungstenAggregate(key=[col0#159], functions=[(sum(col1#160),mode=Complete,isDistinct=false),(avg(col2#161),mode=Final,isDistinct=false)], output=[col0#159,sum(col1)#177,avg(col2)#178]) +- TungstenExchange hashpartitioning(col0#159,200), None +- InMemoryColumnarTableScan [col0#159,col1#160,col2#161], InMemoryRelation [col0#159,col1#160,col2#161], true, 10000, StorageLevel(true, true, false, true, 1), ConvertToUnsafe, None ``` Author: Takeshi YAMAMURO <linguin.m.s@gmail.com> Closes #10896 from maropu/SkipGroupbySpike.
Diffstat (limited to 'sql/hive-thriftserver')
0 files changed, 0 insertions, 0 deletions