diff options
author | Dongjoon Hyun <dongjoon@apache.org> | 2016-06-23 11:07:34 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2016-06-23 11:07:34 +0100 |
commit | 5eef1e6c6a8b6202fc6db4a90c4caab5169e86c6 (patch) | |
tree | 70eff7993a955e5c37aa2fcea4b05693baf9b902 /core/src/test/scala | |
parent | 4374a46bfc52ee4f3ae9f61ccedc77a62aa9d4ee (diff) | |
download | spark-5eef1e6c6a8b6202fc6db4a90c4caab5169e86c6.tar.gz spark-5eef1e6c6a8b6202fc6db4a90c4caab5169e86c6.tar.bz2 spark-5eef1e6c6a8b6202fc6db4a90c4caab5169e86c6.zip |
[SPARK-15660][CORE] Update RDD `variance/stdev` description and add popVariance/popStdev
## What changes were proposed in this pull request?
In Spark-11490, `variance/stdev` are redefined as the **sample** `variance/stdev` instead of population ones. This PR updates the other old documentations to prevent users from misunderstanding. This will update the following Scala/Java API docs.
- http://spark.apache.org/docs/2.0.0-preview/api/scala/index.html#org.apache.spark.api.java.JavaDoubleRDD
- http://spark.apache.org/docs/2.0.0-preview/api/scala/index.html#org.apache.spark.rdd.DoubleRDDFunctions
- http://spark.apache.org/docs/2.0.0-preview/api/scala/index.html#org.apache.spark.util.StatCounter
- http://spark.apache.org/docs/2.0.0-preview/api/java/org/apache/spark/api/java/JavaDoubleRDD.html
- http://spark.apache.org/docs/2.0.0-preview/api/java/org/apache/spark/rdd/DoubleRDDFunctions.html
- http://spark.apache.org/docs/2.0.0-preview/api/java/org/apache/spark/util/StatCounter.html
Also, this PR adds them `popVariance` and `popStdev` functions clearly.
## How was this patch tested?
Pass the updated Jenkins tests.
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #13403 from dongjoon-hyun/SPARK-15660.
Diffstat (limited to 'core/src/test/scala')
-rw-r--r-- | core/src/test/scala/org/apache/spark/PartitioningSuite.scala | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/core/src/test/scala/org/apache/spark/PartitioningSuite.scala b/core/src/test/scala/org/apache/spark/PartitioningSuite.scala index 3d31c7864e..c5d4968ef7 100644 --- a/core/src/test/scala/org/apache/spark/PartitioningSuite.scala +++ b/core/src/test/scala/org/apache/spark/PartitioningSuite.scala @@ -244,6 +244,10 @@ class PartitioningSuite extends SparkFunSuite with SharedSparkContext with Priva assert(abs(6.0/2 - rdd.mean) < 0.01) assert(abs(1.0 - rdd.variance) < 0.01) assert(abs(1.0 - rdd.stdev) < 0.01) + assert(abs(rdd.variance - rdd.popVariance) < 1e-14) + assert(abs(rdd.stdev - rdd.popStdev) < 1e-14) + assert(abs(2.0 - rdd.sampleVariance) < 1e-14) + assert(abs(Math.sqrt(2.0) - rdd.sampleStdev) < 1e-14) assert(stats.max === 4.0) assert(stats.min === 2.0) |