diff options
| author | Reynold Xin <rxin@apache.org> | 2013-12-17 22:26:21 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@apache.org> | 2013-12-17 22:26:21 -0800 |
| commit | f4effb375e93993be1777ebb423c100ea8422f24 (patch) | |
| tree | a5ab5eae51c0f393efa3a415ee1a255bba9d6924 | |
| parent | 7a8169be9a0b6b3d0d53a98aa38940d47b201296 (diff) | |
| parent | 9a6864d016f0f923c885ca2a4977104cd19ded4f (diff) | |
| download | spark-f4effb375e93993be1777ebb423c100ea8422f24.tar.gz spark-f4effb375e93993be1777ebb423c100ea8422f24.tar.bz2 spark-f4effb375e93993be1777ebb423c100ea8422f24.zip | |
Merge pull request #273 from rxin/top
Fixed a performance problem in RDD.top and BoundedPriorityQueue
BoundedPriority was actually traversing the entire queue to calculate the size, resulting in bad performance in insertion.
This should also cherry pick cleanly into branch-0.8.
| -rw-r--r-- | core/src/main/scala/org/apache/spark/util/BoundedPriorityQueue.scala | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/util/BoundedPriorityQueue.scala b/core/src/main/scala/org/apache/spark/util/BoundedPriorityQueue.scala index 0b51c23f7b..a38329df03 100644 --- a/core/src/main/scala/org/apache/spark/util/BoundedPriorityQueue.scala +++ b/core/src/main/scala/org/apache/spark/util/BoundedPriorityQueue.scala @@ -34,6 +34,8 @@ class BoundedPriorityQueue[A](maxSize: Int)(implicit ord: Ordering[A]) override def iterator: Iterator[A] = underlying.iterator.asScala + override def size: Int = underlying.size + override def ++=(xs: TraversableOnce[A]): this.type = { xs.foreach { this += _ } this |