From d3c90b74edecc527ee468bead41d1cca0b667668 Mon Sep 17 00:00:00 2001 From: Reynold Xin Date: Thu, 1 Dec 2016 21:38:52 -0800 Subject: [SPARK-18663][SQL] Simplify CountMinSketch aggregate implementation ## What changes were proposed in this pull request? SPARK-18429 introduced count-min sketch aggregate function for SQL, but the implementation and testing is more complicated than needed. This simplifies the test cases and removes support for data types that don't have clear equality semantics: 1. Removed support for floating point and decimal types. 2. Removed the heavy randomized tests. The underlying CountMinSketch implementation already had pretty good test coverage through randomized tests, and the SPARK-18429 implementation is just to add an aggregate function wrapper around CountMinSketch. There is no need for randomized tests at three different levels of the implementations. ## How was this patch tested? A lot of the change is to simplify test cases. Author: Reynold Xin Closes #16093 from rxin/SPARK-18663. --- project/MimaExcludes.scala | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) (limited to 'project') diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala index 4995af034f..b113bbf803 100644 --- a/project/MimaExcludes.scala +++ b/project/MimaExcludes.scala @@ -34,6 +34,11 @@ import com.typesafe.tools.mima.core.ProblemFilters._ */ object MimaExcludes { + lazy val v22excludes = v21excludes ++ Seq( + // [SPARK-18663][SQL] Simplify CountMinSketch aggregate implementation + ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.util.sketch.CountMinSketch.toByteArray") + ) + // Exclude rules for 2.1.x lazy val v21excludes = v20excludes ++ { Seq( @@ -912,7 +917,8 @@ object MimaExcludes { } def excludes(version: String) = version match { - case v if v.startsWith("2.1") => v21excludes + case v if v.startsWith("2.2") => v22excludes + case v if v.startsWith("2.1") => v22excludes // TODO: Update this when we bump version to 2.2 case v if v.startsWith("2.0") => v20excludes case _ => Seq() } -- cgit v1.2.3