diff options
| author | DB Tsai <dbtsai@dbtsai.com> | 2014-07-11 23:04:43 -0700 |
|---|---|---|
| committer | Xiangrui Meng <meng@databricks.com> | 2014-07-11 23:04:43 -0700 |
| commit | 55960869358d4f8aa5b2e3b17d87b0b02ba9acdd (patch) | |
| tree | 951cd9861e44f12f5c21f58f305dcfa3caa5ee38 /project | |
| parent | cbff18774b0a2f346901ddf2f566be50561a57c7 (diff) | |
| download | spark-55960869358d4f8aa5b2e3b17d87b0b02ba9acdd.tar.gz spark-55960869358d4f8aa5b2e3b17d87b0b02ba9acdd.tar.bz2 spark-55960869358d4f8aa5b2e3b17d87b0b02ba9acdd.zip | |
[SPARK-1969][MLlib] Online summarizer APIs for mean, variance, min, and max
It basically moved the private ColumnStatisticsAggregator class from RowMatrix to public available DeveloperApi with documentation and unitests.
Changes:
1) Moved the private implementation from org.apache.spark.mllib.linalg.ColumnStatisticsAggregator to org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
2) When creating OnlineSummarizer object, the number of columns is not needed in the constructor. It's determined when users add the first sample.
3) Added the APIs documentation for MultivariateOnlineSummarizer.
4) Added the unittests for MultivariateOnlineSummarizer.
Author: DB Tsai <dbtsai@dbtsai.com>
Closes #955 from dbtsai/dbtsai-summarizer and squashes the following commits:
b13ac90 [DB Tsai] dbtsai-summarizer
Diffstat (limited to 'project')
| -rw-r--r-- | project/MimaExcludes.scala | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala index 3b7b87b80c..d67c6571a0 100644 --- a/project/MimaExcludes.scala +++ b/project/MimaExcludes.scala @@ -75,6 +75,7 @@ object MimaExcludes { ProblemFilters.exclude[MissingMethodProblem]( "org.apache.spark.mllib.recommendation.ALS.org$apache$spark$mllib$recommendation$ALS$$<init>$default$7") ) ++ + MimaBuild.excludeSparkClass("mllib.linalg.distributed.ColumnStatisticsAggregator") ++ MimaBuild.excludeSparkClass("rdd.ZippedRDD") ++ MimaBuild.excludeSparkClass("rdd.ZippedPartition") ++ MimaBuild.excludeSparkClass("util.SerializableHyperLogLog") ++ |