diff options
author | Burak Yavuz <brkyvz@gmail.com> | 2015-05-03 21:44:39 -0700 |
---|---|---|
committer | Reynold Xin <rxin@databricks.com> | 2015-05-03 21:44:39 -0700 |
commit | 9646018bb4466433521b4e602b808f16e8d0ffdb (patch) | |
tree | fe4df54135dcc0806af5f801c4753d9b88dbcc16 /sql/core/src/test/java | |
parent | 1ffa8cb91f8badf12a8aa190dc25920715a00db7 (diff) | |
download | spark-9646018bb4466433521b4e602b808f16e8d0ffdb.tar.gz spark-9646018bb4466433521b4e602b808f16e8d0ffdb.tar.bz2 spark-9646018bb4466433521b4e602b808f16e8d0ffdb.zip |
[SPARK-7241] Pearson correlation for DataFrames
submitting this PR from a phone, excuse the brevity.
adds Pearson correlation to Dataframes, reusing the covariance calculation code
cc mengxr rxin
Author: Burak Yavuz <brkyvz@gmail.com>
Closes #5858 from brkyvz/df-corr and squashes the following commits:
285b838 [Burak Yavuz] addressed comments v2.0
d10babb [Burak Yavuz] addressed comments v0.2
4b74b24 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into df-corr
4fe693b [Burak Yavuz] addressed comments v0.1
a682d06 [Burak Yavuz] ready for PR
Diffstat (limited to 'sql/core/src/test/java')
-rw-r--r-- | sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java | 7 |
1 files changed, 7 insertions, 0 deletions
diff --git a/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java b/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java index 96fe66d0b8..78e847239f 100644 --- a/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java +++ b/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java @@ -188,6 +188,13 @@ public class JavaDataFrameSuite { } @Test + public void testCorrelation() { + DataFrame df = context.table("testData2"); + Double pearsonCorr = df.stat().corr("a", "b", "pearson"); + Assert.assertTrue(Math.abs(pearsonCorr) < 1e-6); + } + + @Test public void testCovariance() { DataFrame df = context.table("testData2"); Double result = df.stat().cov("a", "b"); |