aboutsummaryrefslogtreecommitdiff
path: root/project
diff options
context:
space:
mode:
authorReynold Xin <rxin@databricks.com>2016-12-01 21:38:52 -0800
committerReynold Xin <rxin@databricks.com>2016-12-01 21:38:52 -0800
commitd3c90b74edecc527ee468bead41d1cca0b667668 (patch)
tree1b64571522c38155e472e0da58dac55907a22225 /project
parenta5f02b00291e0a22429a3dca81f12cf6d38fea0b (diff)
downloadspark-d3c90b74edecc527ee468bead41d1cca0b667668.tar.gz
spark-d3c90b74edecc527ee468bead41d1cca0b667668.tar.bz2
spark-d3c90b74edecc527ee468bead41d1cca0b667668.zip
[SPARK-18663][SQL] Simplify CountMinSketch aggregate implementation
## What changes were proposed in this pull request? SPARK-18429 introduced count-min sketch aggregate function for SQL, but the implementation and testing is more complicated than needed. This simplifies the test cases and removes support for data types that don't have clear equality semantics: 1. Removed support for floating point and decimal types. 2. Removed the heavy randomized tests. The underlying CountMinSketch implementation already had pretty good test coverage through randomized tests, and the SPARK-18429 implementation is just to add an aggregate function wrapper around CountMinSketch. There is no need for randomized tests at three different levels of the implementations. ## How was this patch tested? A lot of the change is to simplify test cases. Author: Reynold Xin <rxin@databricks.com> Closes #16093 from rxin/SPARK-18663.
Diffstat (limited to 'project')
-rw-r--r--project/MimaExcludes.scala8
1 files changed, 7 insertions, 1 deletions
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index 4995af034f..b113bbf803 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -34,6 +34,11 @@ import com.typesafe.tools.mima.core.ProblemFilters._
*/
object MimaExcludes {
+ lazy val v22excludes = v21excludes ++ Seq(
+ // [SPARK-18663][SQL] Simplify CountMinSketch aggregate implementation
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.util.sketch.CountMinSketch.toByteArray")
+ )
+
// Exclude rules for 2.1.x
lazy val v21excludes = v20excludes ++ {
Seq(
@@ -912,7 +917,8 @@ object MimaExcludes {
}
def excludes(version: String) = version match {
- case v if v.startsWith("2.1") => v21excludes
+ case v if v.startsWith("2.2") => v22excludes
+ case v if v.startsWith("2.1") => v22excludes // TODO: Update this when we bump version to 2.2
case v if v.startsWith("2.0") => v20excludes
case _ => Seq()
}