diff options
| author | Xiao Li <gatorsmile@gmail.com> | 2017-03-08 09:36:01 -0800 |
|---|---|---|
| committer | Xiao Li <gatorsmile@gmail.com> | 2017-03-08 09:36:01 -0800 |
| commit | 9a6ac7226fd09d570cae08d0daea82d9bca189a0 (patch) | |
| tree | 19d31d8de6fd26ad6f3168891d92b693e19fa802 /R/pkg | |
| parent | 5f7d835d380c1a558a4a6d8366140cd96ee202eb (diff) | |
| download | spark-9a6ac7226fd09d570cae08d0daea82d9bca189a0.tar.gz spark-9a6ac7226fd09d570cae08d0daea82d9bca189a0.tar.bz2 spark-9a6ac7226fd09d570cae08d0daea82d9bca189a0.zip | |
[SPARK-19601][SQL] Fix CollapseRepartition rule to preserve shuffle-enabled Repartition
### What changes were proposed in this pull request?
Observed by felixcheung in https://github.com/apache/spark/pull/16739, when users use the shuffle-enabled `repartition` API, they expect the partition they got should be the exact number they provided, even if they call shuffle-disabled `coalesce` later.
Currently, `CollapseRepartition` rule does not consider whether shuffle is enabled or not. Thus, we got the following unexpected result.
```Scala
val df = spark.range(0, 10000, 1, 5)
val df2 = df.repartition(10)
assert(df2.coalesce(13).rdd.getNumPartitions == 5)
assert(df2.coalesce(7).rdd.getNumPartitions == 5)
assert(df2.coalesce(3).rdd.getNumPartitions == 3)
```
This PR is to fix the issue. We preserve shuffle-enabled Repartition.
### How was this patch tested?
Added a test case
Author: Xiao Li <gatorsmile@gmail.com>
Closes #16933 from gatorsmile/CollapseRepartition.
Diffstat (limited to 'R/pkg')
| -rw-r--r-- | R/pkg/inst/tests/testthat/test_sparkSQL.R | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/R/pkg/inst/tests/testthat/test_sparkSQL.R b/R/pkg/inst/tests/testthat/test_sparkSQL.R index 620b633637..9735fe3201 100644 --- a/R/pkg/inst/tests/testthat/test_sparkSQL.R +++ b/R/pkg/inst/tests/testthat/test_sparkSQL.R @@ -2592,8 +2592,8 @@ test_that("coalesce, repartition, numPartitions", { df2 <- repartition(df1, 10) expect_equal(getNumPartitions(df2), 10) - expect_equal(getNumPartitions(coalesce(df2, 13)), 5) - expect_equal(getNumPartitions(coalesce(df2, 7)), 5) + expect_equal(getNumPartitions(coalesce(df2, 13)), 10) + expect_equal(getNumPartitions(coalesce(df2, 7)), 7) expect_equal(getNumPartitions(coalesce(df2, 3)), 3) }) |