diff options
| author | li-zhihui <zhihui.li@intel.com> | 2014-08-08 22:52:56 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-08-08 22:53:14 -0700 |
| commit | 3311da2f9efc5ff2c7d01273ac08f719b067d11d (patch) | |
| tree | 9276ba7419042fc2fd5ec4949647f220e539cf47 /docs/configuration.md | |
| parent | dd11e4e4253897a18d3bb50f8293a580cbe578b3 (diff) | |
| download | spark-3311da2f9efc5ff2c7d01273ac08f719b067d11d.tar.gz spark-3311da2f9efc5ff2c7d01273ac08f719b067d11d.tar.bz2 spark-3311da2f9efc5ff2c7d01273ac08f719b067d11d.zip | |
[SPARK-2635] Fix race condition at SchedulerBackend.isReady in standalone mode
In SPARK-1946(PR #900), configuration <code>spark.scheduler.minRegisteredExecutorsRatio</code> was introduced. However, in standalone mode, there is a race condition where isReady() can return true because totalExpectedExecutors has not been correctly set.
Because expected executors is uncertain in standalone mode, the PR try to use CPU cores(<code>--total-executor-cores</code>) as expected resources to judge whether SchedulerBackend is ready.
Author: li-zhihui <zhihui.li@intel.com>
Author: Li Zhihui <zhihui.li@intel.com>
Closes #1525 from li-zhihui/fixre4s and squashes the following commits:
e9a630b [Li Zhihui] Rename variable totalExecutors and clean codes
abf4860 [Li Zhihui] Push down variable totalExpectedResources to children classes
ca54bd9 [li-zhihui] Format log with String interpolation
88c7dc6 [li-zhihui] Few codes and docs refactor
41cf47e [li-zhihui] Fix race condition at SchedulerBackend.isReady in standalone mode
(cherry picked from commit 28dbae85aaf6842e22cd7465cb11cb34d58fc56d)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 13 |
1 files changed, 7 insertions, 6 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 4d27c5a918..617a72a021 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -825,21 +825,22 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> </tr> - <td><code>spark.scheduler.minRegisteredExecutorsRatio</code></td> + <td><code>spark.scheduler.minRegisteredResourcesRatio</code></td> <td>0</td> <td> - The minimum ratio of registered executors (registered executors / total expected executors) + The minimum ratio of registered resources (registered resources / total expected resources) + (resources are executors in yarn mode, CPU cores in standalone mode) to wait for before scheduling begins. Specified as a double between 0 and 1. - Regardless of whether the minimum ratio of executors has been reached, + Regardless of whether the minimum ratio of resources has been reached, the maximum amount of time it will wait before scheduling begins is controlled by config - <code>spark.scheduler.maxRegisteredExecutorsWaitingTime</code> + <code>spark.scheduler.maxRegisteredResourcesWaitingTime</code> </td> </tr> <tr> - <td><code>spark.scheduler.maxRegisteredExecutorsWaitingTime</code></td> + <td><code>spark.scheduler.maxRegisteredResourcesWaitingTime</code></td> <td>30000</td> <td> - Maximum amount of time to wait for executors to register before scheduling begins + Maximum amount of time to wait for resources to register before scheduling begins (in milliseconds). </td> </tr> |