diff options
| author | Reynold Xin <rxin@databricks.com> | 2016-04-09 00:00:39 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-04-09 00:00:39 -0700 |
| commit | 520dde48d0d52dbbbbe1710a3275fdd5355dd69d (patch) | |
| tree | c6fdf831b43c044e894f5cf24af30650e6aa82c0 /project | |
| parent | 2f0b882e5c8787b09bedcc8208e6dcc5662dbbab (diff) | |
| download | spark-520dde48d0d52dbbbbe1710a3275fdd5355dd69d.tar.gz spark-520dde48d0d52dbbbbe1710a3275fdd5355dd69d.tar.bz2 spark-520dde48d0d52dbbbbe1710a3275fdd5355dd69d.zip | |
[SPARK-14451][SQL] Move encoder definition into Aggregator interface
## What changes were proposed in this pull request?
When we first introduced Aggregators, we required the user of Aggregators to (implicitly) specify the encoders. It would actually make more sense to have the encoders be specified by the implementation of Aggregators, since each implementation should have the most state about how to encode its own data type.
Note that this simplifies the Java API because Java users no longer need to explicitly specify encoders for aggregators.
## How was this patch tested?
Updated unit tests.
Author: Reynold Xin <rxin@databricks.com>
Closes #12231 from rxin/SPARK-14451.
Diffstat (limited to 'project')
| -rw-r--r-- | project/MimaExcludes.scala | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala index f240c30427..290de794dc 100644 --- a/project/MimaExcludes.scala +++ b/project/MimaExcludes.scala @@ -329,6 +329,11 @@ object MimaExcludes { ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.GroupedDataset"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.Dataset.subtract"), + // [SPARK-14451][SQL] Move encoder definition into Aggregator interface + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.expressions.Aggregator.toColumn"), + ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.expressions.Aggregator.bufferEncoder"), + ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.expressions.Aggregator.outputEncoder"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.mllib.evaluation.MultilabelMetrics.this"), ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.classification.LogisticRegressionSummary.predictions"), ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.ml.classification.LogisticRegressionSummary.predictions") |