diff options
author | Burak Yavuz <brkyvz@gmail.com> | 2015-05-01 13:29:17 -0700 |
---|---|---|
committer | Reynold Xin <rxin@databricks.com> | 2015-05-01 13:29:17 -0700 |
commit | 4dc8d74491b101a794cf8d386d8c5ebc6019b75f (patch) | |
tree | 01733e92623635c80a0e3d7b50869b742f3f82a1 /sql/core/src/test/java | |
parent | 7b5dd3e3c0030087eea5a8224789352c03717c1d (diff) | |
download | spark-4dc8d74491b101a794cf8d386d8c5ebc6019b75f.tar.gz spark-4dc8d74491b101a794cf8d386d8c5ebc6019b75f.tar.bz2 spark-4dc8d74491b101a794cf8d386d8c5ebc6019b75f.zip |
[SPARK-7240][SQL] Single pass covariance calculation for dataframes
Added the calculation of covariance between two columns to DataFrames.
cc mengxr rxin
Author: Burak Yavuz <brkyvz@gmail.com>
Closes #5825 from brkyvz/df-cov and squashes the following commits:
cb18046 [Burak Yavuz] changed to sample covariance
f2e862b [Burak Yavuz] fixed failed test
51e39b8 [Burak Yavuz] moved implementation
0c6a759 [Burak Yavuz] addressed math comments
8456eca [Burak Yavuz] fix pyStyle3
aa2ad29 [Burak Yavuz] fix pyStyle2
4e97a50 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into df-cov
e3b0b85 [Burak Yavuz] addressed comments v0.1
a7115f1 [Burak Yavuz] fix python style
7dc6dbc [Burak Yavuz] reorder imports
408cb77 [Burak Yavuz] initial commit
Diffstat (limited to 'sql/core/src/test/java')
-rw-r--r-- | sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java | 7 |
1 files changed, 7 insertions, 0 deletions
diff --git a/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java b/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java index ebe96e649d..96fe66d0b8 100644 --- a/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java +++ b/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java @@ -186,4 +186,11 @@ public class JavaDataFrameSuite { DataFrame results = df.stat().freqItems(cols, 0.2); Assert.assertTrue(results.collect()[0].getSeq(0).contains(1)); } + + @Test + public void testCovariance() { + DataFrame df = context.table("testData2"); + Double result = df.stat().cov("a", "b"); + Assert.assertTrue(Math.abs(result) < 1e-6); + } } |