diff options
author | Wenchen Fan <wenchen@databricks.com> | 2016-03-08 11:45:08 -0800 |
---|---|---|
committer | Reynold Xin <rxin@databricks.com> | 2016-03-08 11:45:08 -0800 |
commit | 46881b4ea229aecbb481626d8b9fbca24c0df075 (patch) | |
tree | b87768d6d17123041029896afa0ab7b1d6ef06b7 /sql/hive | |
parent | ad3c9a9730535faa6d270ee83412f79ec3db8333 (diff) | |
download | spark-46881b4ea229aecbb481626d8b9fbca24c0df075.tar.gz spark-46881b4ea229aecbb481626d8b9fbca24c0df075.tar.bz2 spark-46881b4ea229aecbb481626d8b9fbca24c0df075.zip |
[SPARK-12727][SQL] support SQL generation for aggregate with multi-distinct
## What changes were proposed in this pull request?
This PR add SQL generation support for aggregate with multi-distinct, by simply moving the `DistinctAggregationRewriter` rule to optimizer.
More discussions are needed as this breaks an import contract: analyzed plan should be able to run without optimization. However, the `ComputeCurrentTime` rule has kind of broken it already, and I think maybe we should add a new phase for this kind of rules, because strictly speaking they don't belong to analysis and is coupled with the physical plan implementation.
## How was this patch tested?
existing tests
Author: Wenchen Fan <wenchen@databricks.com>
Closes #11579 from cloud-fan/distinct.
Diffstat (limited to 'sql/hive')
-rw-r--r-- | sql/hive/src/test/scala/org/apache/spark/sql/hive/LogicalPlanToSQLSuite.scala | 5 |
1 files changed, 1 insertions, 4 deletions
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/LogicalPlanToSQLSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/LogicalPlanToSQLSuite.scala index ed85856f01..ccce9871e2 100644 --- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/LogicalPlanToSQLSuite.scala +++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/LogicalPlanToSQLSuite.scala @@ -139,7 +139,6 @@ class LogicalPlanToSQLSuite extends SQLBuilderTest with SQLTestUtils { """.stripMargin) } - test("intersect") { checkHiveQl("SELECT * FROM t0 INTERSECT SELECT * FROM t0") } @@ -367,9 +366,7 @@ class LogicalPlanToSQLSuite extends SQLBuilderTest with SQLTestUtils { checkHiveQl("SELECT * FROM parquet_t0 TABLESAMPLE(0.1 PERCENT) WHERE 1=0") } - // TODO Enable this - // Query plans transformed by DistinctAggregationRewriter are not recognized yet - ignore("multi-distinct columns") { + test("multi-distinct columns") { checkHiveQl("SELECT a, COUNT(DISTINCT b), COUNT(DISTINCT c), SUM(d) FROM parquet_t2 GROUP BY a") } |