diff options
author | root <root@iZbp1gsnrlfzjxh82cz80vZ.(none)> | 2016-11-08 12:09:32 +0100 |
---|---|---|
committer | Herman van Hovell <hvanhovell@databricks.com> | 2016-11-08 12:09:32 +0100 |
commit | c291bd2745a8a2e4ba91d8697879eb8da10287e2 (patch) | |
tree | 5fd2f31509376493cdbd26a188f986961c880836 /mllib/src/test | |
parent | 47731e1865fa1e3a8881a1f4420017bdc026e455 (diff) | |
download | spark-c291bd2745a8a2e4ba91d8697879eb8da10287e2.tar.gz spark-c291bd2745a8a2e4ba91d8697879eb8da10287e2.tar.bz2 spark-c291bd2745a8a2e4ba91d8697879eb8da10287e2.zip |
[SPARK-18137][SQL] Fix RewriteDistinctAggregates UnresolvedException when a UDAF has a foldable TypeCheck
## What changes were proposed in this pull request?
In RewriteDistinctAggregates rewrite funtion,after the UDAF's childs are mapped to AttributeRefference, If the UDAF(such as ApproximatePercentile) has a foldable TypeCheck for the input, It will failed because the AttributeRefference is not foldable,then the UDAF is not resolved, and then nullify on the unresolved object will throw a Exception.
In this PR, only map Unfoldable child to AttributeRefference, this can avoid the UDAF's foldable TypeCheck. and then only Expand Unfoldable child, there is no need to Expand a static value(foldable value).
**Before sql result**
> select percentile_approxy(key,0.99999),count(distinct key),sume(distinc key) from src limit 1
> org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'percentile_approx(CAST(src.`key` AS DOUBLE), CAST(0.99999BD AS DOUBLE), 10000)
> at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:92)
> at org.apache.spark.sql.catalyst.optimizer.RewriteDistinctAggregates$.org$apache$spark$sql$catalyst$optimizer$RewriteDistinctAggregates$$nullify(RewriteDistinctAggregates.scala:261)
**After sql result**
> select percentile_approxy(key,0.99999),count(distinct key),sume(distinc key) from src limit 1
> [498.0,309,79136]
## How was this patch tested?
Add a test case in HiveUDFSuit.
Author: root <root@iZbp1gsnrlfzjxh82cz80vZ.(none)>
Closes #15668 from windpiger/RewriteDistinctUDAFUnresolveExcep.
Diffstat (limited to 'mllib/src/test')
0 files changed, 0 insertions, 0 deletions