aboutsummaryrefslogtreecommitdiff
path: root/core/src/test/scala
diff options
context:
space:
mode:
authorDongjoon Hyun <dongjoon@apache.org>2016-06-23 11:07:34 +0100
committerSean Owen <sowen@cloudera.com>2016-06-23 11:07:34 +0100
commit5eef1e6c6a8b6202fc6db4a90c4caab5169e86c6 (patch)
tree70eff7993a955e5c37aa2fcea4b05693baf9b902 /core/src/test/scala
parent4374a46bfc52ee4f3ae9f61ccedc77a62aa9d4ee (diff)
downloadspark-5eef1e6c6a8b6202fc6db4a90c4caab5169e86c6.tar.gz
spark-5eef1e6c6a8b6202fc6db4a90c4caab5169e86c6.tar.bz2
spark-5eef1e6c6a8b6202fc6db4a90c4caab5169e86c6.zip
[SPARK-15660][CORE] Update RDD `variance/stdev` description and add popVariance/popStdev
## What changes were proposed in this pull request? In Spark-11490, `variance/stdev` are redefined as the **sample** `variance/stdev` instead of population ones. This PR updates the other old documentations to prevent users from misunderstanding. This will update the following Scala/Java API docs. - http://spark.apache.org/docs/2.0.0-preview/api/scala/index.html#org.apache.spark.api.java.JavaDoubleRDD - http://spark.apache.org/docs/2.0.0-preview/api/scala/index.html#org.apache.spark.rdd.DoubleRDDFunctions - http://spark.apache.org/docs/2.0.0-preview/api/scala/index.html#org.apache.spark.util.StatCounter - http://spark.apache.org/docs/2.0.0-preview/api/java/org/apache/spark/api/java/JavaDoubleRDD.html - http://spark.apache.org/docs/2.0.0-preview/api/java/org/apache/spark/rdd/DoubleRDDFunctions.html - http://spark.apache.org/docs/2.0.0-preview/api/java/org/apache/spark/util/StatCounter.html Also, this PR adds them `popVariance` and `popStdev` functions clearly. ## How was this patch tested? Pass the updated Jenkins tests. Author: Dongjoon Hyun <dongjoon@apache.org> Closes #13403 from dongjoon-hyun/SPARK-15660.
Diffstat (limited to 'core/src/test/scala')
-rw-r--r--core/src/test/scala/org/apache/spark/PartitioningSuite.scala4
1 files changed, 4 insertions, 0 deletions
diff --git a/core/src/test/scala/org/apache/spark/PartitioningSuite.scala b/core/src/test/scala/org/apache/spark/PartitioningSuite.scala
index 3d31c7864e..c5d4968ef7 100644
--- a/core/src/test/scala/org/apache/spark/PartitioningSuite.scala
+++ b/core/src/test/scala/org/apache/spark/PartitioningSuite.scala
@@ -244,6 +244,10 @@ class PartitioningSuite extends SparkFunSuite with SharedSparkContext with Priva
assert(abs(6.0/2 - rdd.mean) < 0.01)
assert(abs(1.0 - rdd.variance) < 0.01)
assert(abs(1.0 - rdd.stdev) < 0.01)
+ assert(abs(rdd.variance - rdd.popVariance) < 1e-14)
+ assert(abs(rdd.stdev - rdd.popStdev) < 1e-14)
+ assert(abs(2.0 - rdd.sampleVariance) < 1e-14)
+ assert(abs(Math.sqrt(2.0) - rdd.sampleStdev) < 1e-14)
assert(stats.max === 4.0)
assert(stats.min === 2.0)