diff options
| author | Perinkulam I. Ganesh <gip@us.ibm.com> | 2015-07-23 07:46:20 +0100 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2015-07-23 07:47:42 +0100 |
| commit | b983d493b490ca8bafe7eb988b62a250987ae353 (patch) | |
| tree | 416e19d427010b3cab6f8ab0b27f68038971abe0 /core | |
| parent | 6d0d8b406942edcf9fc97e76fb227ff1eb35ca3a (diff) | |
| download | spark-b983d493b490ca8bafe7eb988b62a250987ae353.tar.gz spark-b983d493b490ca8bafe7eb988b62a250987ae353.tar.bz2 spark-b983d493b490ca8bafe7eb988b62a250987ae353.zip | |
[SPARK-8695] [CORE] [MLLIB] TreeAggregation shouldn't be triggered when it doesn't save wall-clock time.
Author: Perinkulam I. Ganesh <gip@us.ibm.com>
Closes #7397 from piganesh/SPARK-8695 and squashes the following commits:
041620c [Perinkulam I. Ganesh] [SPARK-8695][CORE][MLlib] TreeAggregation shouldn't be triggered when it doesn't save wall-clock time.
9ad067c [Perinkulam I. Ganesh] [SPARK-8695] [core] [WIP] TreeAggregation shouldn't be triggered for 5 partitions
a6fed07 [Perinkulam I. Ganesh] [SPARK-8695] [core] [WIP] TreeAggregation shouldn't be triggered for 5 partitions
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/rdd/RDD.scala | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/rdd/RDD.scala b/core/src/main/scala/org/apache/spark/rdd/RDD.scala index 394c6686cb..6d61d22738 100644 --- a/core/src/main/scala/org/apache/spark/rdd/RDD.scala +++ b/core/src/main/scala/org/apache/spark/rdd/RDD.scala @@ -1082,7 +1082,9 @@ abstract class RDD[T: ClassTag]( val scale = math.max(math.ceil(math.pow(numPartitions, 1.0 / depth)).toInt, 2) // If creating an extra level doesn't help reduce // the wall-clock time, we stop tree aggregation. - while (numPartitions > scale + numPartitions / scale) { + + // Don't trigger TreeAggregation when it doesn't save wall-clock time + while (numPartitions > scale + math.ceil(numPartitions.toDouble / scale)) { numPartitions /= scale val curNumPartitions = numPartitions partiallyAggregated = partiallyAggregated.mapPartitionsWithIndex { |