diff options
| author | Sun Rui <rui.sun@intel.com> | 2016-03-28 21:51:02 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-03-28 21:51:02 -0700 |
| commit | d3638d7bffd4ee43db594c0669d86fb64d448fc8 (patch) | |
| tree | 8d636f4edb6c1b34ee92a85be2fa188f87cf3fa8 /R | |
| parent | a180286b7994f9f9a56b84903cc9ee6057ba6624 (diff) | |
| download | spark-d3638d7bffd4ee43db594c0669d86fb64d448fc8.tar.gz spark-d3638d7bffd4ee43db594c0669d86fb64d448fc8.tar.bz2 spark-d3638d7bffd4ee43db594c0669d86fb64d448fc8.zip | |
[SPARK-12792] [SPARKR] Refactor RRDD to support R UDF.
## What changes were proposed in this pull request?
Refactor RRDD by separating the common logic interacting with the R worker to a new class RRunner, which can be used to evaluate R UDFs.
Now RRDD relies on RRuner for RDD computation and RRDD could be reomved if we want to remove RDD API in SparkR later.
## How was this patch tested?
dev/lint-r
SparkR unit tests
Author: Sun Rui <rui.sun@intel.com>
Closes #12024 from sun-rui/SPARK-12792_new.
Diffstat (limited to 'R')
| -rw-r--r-- | R/pkg/inst/tests/testthat/test_rdd.R | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/R/pkg/inst/tests/testthat/test_rdd.R b/R/pkg/inst/tests/testthat/test_rdd.R index 3b0c16be5a..b6c8e1dc6c 100644 --- a/R/pkg/inst/tests/testthat/test_rdd.R +++ b/R/pkg/inst/tests/testthat/test_rdd.R @@ -791,3 +791,11 @@ test_that("sampleByKey() on pairwise RDDs", { expect_equal(lookup(sample, 3)[which.min(lookup(sample, 3))] >= 0, TRUE) expect_equal(lookup(sample, 3)[which.max(lookup(sample, 3))] <= 2000, TRUE) }) + +test_that("Test correct concurrency of RRDD.compute()", { + rdd <- parallelize(sc, 1:1000, 100) + jrdd <- getJRDD(lapply(rdd, function(x) { x }), "row") + zrdd <- callJMethod(jrdd, "zip", jrdd) + count <- callJMethod(zrdd, "count") + expect_equal(count, 1000) +}) |