diff options
| author | Felix Cheung <felixcheung_m@hotmail.com> | 2017-02-15 10:45:37 -0800 |
|---|---|---|
| committer | Felix Cheung <felixcheung@apache.org> | 2017-02-15 10:45:37 -0800 |
| commit | 671bc08ed502815bfa2254c30d64149402acb0c7 (patch) | |
| tree | 3edcf2548e8f58a6a27db9c16050a3ff1d8ae261 /core/src/main/scala/org/apache | |
| parent | c97f4e17de0ce39e8172a5a4ae81f1914816a358 (diff) | |
| download | spark-671bc08ed502815bfa2254c30d64149402acb0c7.tar.gz spark-671bc08ed502815bfa2254c30d64149402acb0c7.tar.bz2 spark-671bc08ed502815bfa2254c30d64149402acb0c7.zip | |
[SPARK-19399][SPARKR] Add R coalesce API for DataFrame and Column
## What changes were proposed in this pull request?
Add coalesce on DataFrame for down partitioning without shuffle and coalesce on Column
## How was this patch tested?
manual, unit tests
Author: Felix Cheung <felixcheung_m@hotmail.com>
Closes #16739 from felixcheung/rcoalesce.
Diffstat (limited to 'core/src/main/scala/org/apache')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/rdd/RDD.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/rdd/RDD.scala b/core/src/main/scala/org/apache/spark/rdd/RDD.scala index 0359508c00..e524675332 100644 --- a/core/src/main/scala/org/apache/spark/rdd/RDD.scala +++ b/core/src/main/scala/org/apache/spark/rdd/RDD.scala @@ -423,7 +423,8 @@ abstract class RDD[T: ClassTag]( * * This results in a narrow dependency, e.g. if you go from 1000 partitions * to 100 partitions, there will not be a shuffle, instead each of the 100 - * new partitions will claim 10 of the current partitions. + * new partitions will claim 10 of the current partitions. If a larger number + * of partitions is requested, it will stay at the current number of partitions. * * However, if you're doing a drastic coalesce, e.g. to numPartitions = 1, * this may result in your computation taking place on fewer nodes than |