diff options
| author | Akash Mishra <akash.mishra20@gmail.com> | 2015-09-10 12:03:11 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-09-10 12:04:02 -0700 |
| commit | a5ef2d0600d5e23ca05fabc1005bb81e5ada0727 (patch) | |
| tree | 7f022cbfa90d70c930da9a4251bc1c19ab536f5d | |
| parent | f0562e8cdbab7ce40f3186da98595312252f8b5c (diff) | |
| download | spark-a5ef2d0600d5e23ca05fabc1005bb81e5ada0727.tar.gz spark-a5ef2d0600d5e23ca05fabc1005bb81e5ada0727.tar.bz2 spark-a5ef2d0600d5e23ca05fabc1005bb81e5ada0727.zip | |
[SPARK-10514] [MESOS] waiting for min no of total cores acquired by Spark by implementing the sufficientResourcesRegistered method
spark.scheduler.minRegisteredResourcesRatio configuration parameter works for YARN mode but not for Mesos Coarse grained mode.
If the parameter specified default value of 0 will be set for spark.scheduler.minRegisteredResourcesRatio in base class and this method will always return true.
There are no existing test for YARN mode too. Hence not added test for the same.
Author: Akash Mishra <akash.mishra20@gmail.com>
Closes #8672 from SleepyThread/master.
| -rw-r--r-- | core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala | 4 | ||||
| -rw-r--r-- | docs/configuration.md | 5 |
2 files changed, 7 insertions, 2 deletions
diff --git a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala index 65df887477..65cb5016cf 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala @@ -222,6 +222,10 @@ private[spark] class CoarseMesosSchedulerBackend( markRegistered() } + override def sufficientResourcesRegistered(): Boolean = { + totalCoresAcquired >= maxCores * minRegisteredRatio + } + override def disconnected(d: SchedulerDriver) {} override def reregistered(d: SchedulerDriver, masterInfo: MasterInfo) {} diff --git a/docs/configuration.md b/docs/configuration.md index 0b1a273916..1a701f1888 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1112,10 +1112,11 @@ Apart from these, the following properties are also available, and may be useful </tr> <tr> <td><code>spark.scheduler.minRegisteredResourcesRatio</code></td> - <td>0.8 for YARN mode; 0.0 otherwise</td> + <td>0.8 for YARN mode; 0.0 for standalone mode and Mesos coarse-grained mode</td> <td> The minimum ratio of registered resources (registered resources / total expected resources) - (resources are executors in yarn mode, CPU cores in standalone mode) + (resources are executors in yarn mode, CPU cores in standalone mode and Mesos coarsed-grained + mode ['spark.cores.max' value is total expected resources for Mesos coarse-grained mode] ) to wait for before scheduling begins. Specified as a double between 0.0 and 1.0. Regardless of whether the minimum ratio of resources has been reached, the maximum amount of time it will wait before scheduling begins is controlled by config |