diff options
| author | Xiao Li <gatorsmile@gmail.com> | 2017-03-08 09:36:01 -0800 |
|---|---|---|
| committer | Xiao Li <gatorsmile@gmail.com> | 2017-03-08 09:36:01 -0800 |
| commit | 9a6ac7226fd09d570cae08d0daea82d9bca189a0 (patch) | |
| tree | 19d31d8de6fd26ad6f3168891d92b693e19fa802 /common | |
| parent | 5f7d835d380c1a558a4a6d8366140cd96ee202eb (diff) | |
| download | spark-9a6ac7226fd09d570cae08d0daea82d9bca189a0.tar.gz spark-9a6ac7226fd09d570cae08d0daea82d9bca189a0.tar.bz2 spark-9a6ac7226fd09d570cae08d0daea82d9bca189a0.zip | |
[SPARK-19601][SQL] Fix CollapseRepartition rule to preserve shuffle-enabled Repartition
### What changes were proposed in this pull request?
Observed by felixcheung in https://github.com/apache/spark/pull/16739, when users use the shuffle-enabled `repartition` API, they expect the partition they got should be the exact number they provided, even if they call shuffle-disabled `coalesce` later.
Currently, `CollapseRepartition` rule does not consider whether shuffle is enabled or not. Thus, we got the following unexpected result.
```Scala
val df = spark.range(0, 10000, 1, 5)
val df2 = df.repartition(10)
assert(df2.coalesce(13).rdd.getNumPartitions == 5)
assert(df2.coalesce(7).rdd.getNumPartitions == 5)
assert(df2.coalesce(3).rdd.getNumPartitions == 3)
```
This PR is to fix the issue. We preserve shuffle-enabled Repartition.
### How was this patch tested?
Added a test case
Author: Xiao Li <gatorsmile@gmail.com>
Closes #16933 from gatorsmile/CollapseRepartition.
Diffstat (limited to 'common')
0 files changed, 0 insertions, 0 deletions