diff options
| author | Imran Rashid <irashid@cloudera.com> | 2016-09-29 15:36:40 -0400 |
|---|---|---|
| committer | Andrew Or <andrewor14@gmail.com> | 2016-09-29 15:36:40 -0400 |
| commit | 7f779e7439127efa0e3611f7745e1c8423845198 (patch) | |
| tree | 2f9b734d0a9b310c5b623d519258abfb4d850132 /core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala | |
| parent | 958200497affb40f05e321c2b0e252d365ae02f4 (diff) | |
| download | spark-7f779e7439127efa0e3611f7745e1c8423845198.tar.gz spark-7f779e7439127efa0e3611f7745e1c8423845198.tar.bz2 spark-7f779e7439127efa0e3611f7745e1c8423845198.zip | |
[SPARK-17648][CORE] TaskScheduler really needs offers to be an IndexedSeq
## What changes were proposed in this pull request?
The Seq[WorkerOffer] is accessed by index, so it really should be an
IndexedSeq, otherwise an O(n) operation becomes O(n^2). In practice
this hasn't been an issue b/c where these offers are generated, the call
to `.toSeq` just happens to create an IndexedSeq anyway.I got bitten by
this in performance tests I was doing, and its better for the types to be
more precise so eg. a change in Scala doesn't destroy performance.
## How was this patch tested?
Unit tests via jenkins.
Author: Imran Rashid <irashid@cloudera.com>
Closes #15221 from squito/SPARK-17648.
Diffstat (limited to 'core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala b/core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala index e386052814..7a73e8ed8a 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala @@ -81,7 +81,7 @@ private[spark] class LocalEndpoint( } def reviveOffers() { - val offers = Seq(new WorkerOffer(localExecutorId, localExecutorHostname, freeCores)) + val offers = IndexedSeq(new WorkerOffer(localExecutorId, localExecutorHostname, freeCores)) for (task <- scheduler.resourceOffers(offers).flatten) { freeCores -= scheduler.CPUS_PER_TASK executor.launchTask(executorBackend, taskId = task.taskId, attemptNumber = task.attemptNumber, |