diff options
author | Akash Mishra <akash.mishra20@gmail.com> | 2015-09-10 12:03:11 -0700 |
---|---|---|
committer | Andrew Or <andrew@databricks.com> | 2015-09-10 12:04:02 -0700 |
commit | a5ef2d0600d5e23ca05fabc1005bb81e5ada0727 (patch) | |
tree | 7f022cbfa90d70c930da9a4251bc1c19ab536f5d /core | |
parent | f0562e8cdbab7ce40f3186da98595312252f8b5c (diff) | |
download | spark-a5ef2d0600d5e23ca05fabc1005bb81e5ada0727.tar.gz spark-a5ef2d0600d5e23ca05fabc1005bb81e5ada0727.tar.bz2 spark-a5ef2d0600d5e23ca05fabc1005bb81e5ada0727.zip |
[SPARK-10514] [MESOS] waiting for min no of total cores acquired by Spark by implementing the sufficientResourcesRegistered method
spark.scheduler.minRegisteredResourcesRatio configuration parameter works for YARN mode but not for Mesos Coarse grained mode.
If the parameter specified default value of 0 will be set for spark.scheduler.minRegisteredResourcesRatio in base class and this method will always return true.
There are no existing test for YARN mode too. Hence not added test for the same.
Author: Akash Mishra <akash.mishra20@gmail.com>
Closes #8672 from SleepyThread/master.
Diffstat (limited to 'core')
-rw-r--r-- | core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala index 65df887477..65cb5016cf 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala @@ -222,6 +222,10 @@ private[spark] class CoarseMesosSchedulerBackend( markRegistered() } + override def sufficientResourcesRegistered(): Boolean = { + totalCoresAcquired >= maxCores * minRegisteredRatio + } + override def disconnected(d: SchedulerDriver) {} override def reregistered(d: SchedulerDriver, masterInfo: MasterInfo) {} |