diff options
| author | Sun Rui <rui.sun@intel.com> | 2016-03-28 10:14:28 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-03-28 10:14:28 -0700 |
| commit | 40984f67065eeaea731940008e6677c2323dda3e (patch) | |
| tree | a089e13d0505e01db17d585ef2626071d7a75192 /R/pkg/inst | |
| parent | 68c0c460bfc51d7f69d09b613c49c212dd0b375c (diff) | |
| download | spark-40984f67065eeaea731940008e6677c2323dda3e.tar.gz spark-40984f67065eeaea731940008e6677c2323dda3e.tar.bz2 spark-40984f67065eeaea731940008e6677c2323dda3e.zip | |
[SPARK-12792] [SPARKR] Refactor RRDD to support R UDF.
Refactor RRDD by separating the common logic interacting with the R worker to a new class RRunner, which can be used to evaluate R UDFs.
Now RRDD relies on RRuner for RDD computation and RRDD could be reomved if we want to remove RDD API in SparkR later.
Author: Sun Rui <rui.sun@intel.com>
Closes #10947 from sun-rui/SPARK-12792.
Diffstat (limited to 'R/pkg/inst')
| -rw-r--r-- | R/pkg/inst/tests/testthat/test_rdd.R | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/R/pkg/inst/tests/testthat/test_rdd.R b/R/pkg/inst/tests/testthat/test_rdd.R index 3b0c16be5a..b6c8e1dc6c 100644 --- a/R/pkg/inst/tests/testthat/test_rdd.R +++ b/R/pkg/inst/tests/testthat/test_rdd.R @@ -791,3 +791,11 @@ test_that("sampleByKey() on pairwise RDDs", { expect_equal(lookup(sample, 3)[which.min(lookup(sample, 3))] >= 0, TRUE) expect_equal(lookup(sample, 3)[which.max(lookup(sample, 3))] <= 2000, TRUE) }) + +test_that("Test correct concurrency of RRDD.compute()", { + rdd <- parallelize(sc, 1:1000, 100) + jrdd <- getJRDD(lapply(rdd, function(x) { x }), "row") + zrdd <- callJMethod(jrdd, "zip", jrdd) + count <- callJMethod(zrdd, "count") + expect_equal(count, 1000) +}) |