diff options
author | Giovanni Delussu <surfer@crs4.it> | 2013-03-19 11:25:45 +0100 |
---|---|---|
committer | Giovanni Delussu <surfer@crs4.it> | 2013-03-19 11:25:45 +0100 |
commit | aceae029f7c82f87ed5fc2775190cad1bd6cc326 (patch) | |
tree | ac33bb55634452cf416654701c62c0f495302567 | |
parent | c1e9cdc49f89222b366a14a20ffd937ca0fb9adc (diff) | |
download | spark-aceae029f7c82f87ed5fc2775190cad1bd6cc326.tar.gz spark-aceae029f7c82f87ed5fc2775190cad1bd6cc326.tar.bz2 spark-aceae029f7c82f87ed5fc2775190cad1bd6cc326.zip |
CoalescedRDD changed to work with a big number of partitions both in the original and the new coalesced RDD.
The limitation was in the range that Scala.Int can represent.
-rw-r--r-- | core/src/main/scala/spark/rdd/CoalescedRDD.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/core/src/main/scala/spark/rdd/CoalescedRDD.scala b/core/src/main/scala/spark/rdd/CoalescedRDD.scala index 0d16cf6e85..6d862c0c28 100644 --- a/core/src/main/scala/spark/rdd/CoalescedRDD.scala +++ b/core/src/main/scala/spark/rdd/CoalescedRDD.scala @@ -37,8 +37,8 @@ class CoalescedRDD[T: ClassManifest]( prevSplits.map(_.index).map{idx => new CoalescedRDDPartition(idx, prev, Array(idx)) } } else { (0 until maxPartitions).map { i => - val rangeStart = (i * prevSplits.length) / maxPartitions - val rangeEnd = ((i + 1) * prevSplits.length) / maxPartitions + val rangeStart = ((i.toLong * prevSplits.length) / maxPartitions).toInt + val rangeEnd = (((i.toLong + 1) * prevSplits.length) / maxPartitions).toInt new CoalescedRDDPartition(i, prev, (rangeStart until rangeEnd).toArray) }.toArray } |