diff options
| author | Imran Rashid <irashid@cloudera.com> | 2016-09-29 15:36:40 -0400 |
|---|---|---|
| committer | Andrew Or <andrewor14@gmail.com> | 2016-09-29 15:36:40 -0400 |
| commit | 7f779e7439127efa0e3611f7745e1c8423845198 (patch) | |
| tree | 2f9b734d0a9b310c5b623d519258abfb4d850132 /mesos | |
| parent | 958200497affb40f05e321c2b0e252d365ae02f4 (diff) | |
| download | spark-7f779e7439127efa0e3611f7745e1c8423845198.tar.gz spark-7f779e7439127efa0e3611f7745e1c8423845198.tar.bz2 spark-7f779e7439127efa0e3611f7745e1c8423845198.zip | |
[SPARK-17648][CORE] TaskScheduler really needs offers to be an IndexedSeq
## What changes were proposed in this pull request?
The Seq[WorkerOffer] is accessed by index, so it really should be an
IndexedSeq, otherwise an O(n) operation becomes O(n^2). In practice
this hasn't been an issue b/c where these offers are generated, the call
to `.toSeq` just happens to create an IndexedSeq anyway.I got bitten by
this in performance tests I was doing, and its better for the types to be
more precise so eg. a change in Scala doesn't destroy performance.
## How was this patch tested?
Unit tests via jenkins.
Author: Imran Rashid <irashid@cloudera.com>
Closes #15221 from squito/SPARK-17648.
Diffstat (limited to 'mesos')
2 files changed, 2 insertions, 2 deletions
diff --git a/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala b/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala index eb3b235949..09a252f3c7 100644 --- a/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala +++ b/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala @@ -286,7 +286,7 @@ private[spark] class MesosFineGrainedSchedulerBackend( o.getSlaveId.getValue, o.getHostname, cpus) - } + }.toIndexedSeq val slaveIdToOffer = usableOffers.map(o => o.getSlaveId.getValue -> o).toMap val slaveIdToWorkerOffer = workerOffers.map(o => o.executorId -> o).toMap diff --git a/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackendSuite.scala b/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackendSuite.scala index 7a706ab256..1d7a86f4b0 100644 --- a/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackendSuite.scala +++ b/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackendSuite.scala @@ -283,7 +283,7 @@ class MesosFineGrainedSchedulerBackendSuite mesosOffers2.add(createOffer(1, minMem, minCpu)) reset(taskScheduler) reset(driver) - when(taskScheduler.resourceOffers(any(classOf[Seq[WorkerOffer]]))).thenReturn(Seq(Seq())) + when(taskScheduler.resourceOffers(any(classOf[IndexedSeq[WorkerOffer]]))).thenReturn(Seq(Seq())) when(taskScheduler.CPUS_PER_TASK).thenReturn(2) when(driver.declineOffer(mesosOffers2.get(0).getId)).thenReturn(Status.valueOf(1)) |