diff options
| author | ehnalis <zoltan.zvara@gmail.com> | 2015-05-20 08:27:39 -0500 |
|---|---|---|
| committer | Thomas Graves <tgraves@apache.org> | 2015-05-20 08:27:39 -0500 |
| commit | 3ddf051ee7256f642f8a17768d161c7b5f55c7e1 (patch) | |
| tree | 36d9986102035f54b43ca545cc4c78b2d48b4b34 /docs/running-on-yarn.md | |
| parent | 09265ad7c85c6de6b568ec329daad632d4a79fa3 (diff) | |
| download | spark-3ddf051ee7256f642f8a17768d161c7b5f55c7e1.tar.gz spark-3ddf051ee7256f642f8a17768d161c7b5f55c7e1.tar.bz2 spark-3ddf051ee7256f642f8a17768d161c7b5f55c7e1.zip | |
[SPARK-7533] [YARN] Decrease spacing between AM-RM heartbeats.
Added faster RM-heartbeats on pending container allocations with multiplicative back-off.
Also updated related documentations.
Author: ehnalis <zoltan.zvara@gmail.com>
Closes #6082 from ehnalis/yarn and squashes the following commits:
a1d2101 [ehnalis] MIss-spell fixed.
90f8ba4 [ehnalis] Changed default HB values.
6120295 [ehnalis] Removed the bug, when allocation heartbeat would not start from initial value.
08bac63 [ehnalis] Refined style, grammar, removed duplicated code.
073d283 [ehnalis] [SPARK-7533] [YARN] Decrease spacing between AM-RM heartbeats.
d4408c9 [ehnalis] [SPARK-7533] [YARN] Decrease spacing between AM-RM heartbeats.
Diffstat (limited to 'docs/running-on-yarn.md')
| -rw-r--r-- | docs/running-on-yarn.md | 15 |
1 files changed, 14 insertions, 1 deletions
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 51c1339165..9d55f435e8 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -71,9 +71,22 @@ Most of the configs are the same for Spark on YARN as for other deployment modes </tr> <tr> <td><code>spark.yarn.scheduler.heartbeat.interval-ms</code></td> - <td>5000</td> + <td>3000</td> <td> The interval in ms in which the Spark application master heartbeats into the YARN ResourceManager. + The value is capped at half the value of YARN's configuration for the expiry interval + (<code>yarn.am.liveness-monitor.expiry-interval-ms</code>). + </td> +</tr> +<tr> + <td><code>spark.yarn.scheduler.initial-allocation.interval</code></td> + <td>200ms</td> + <td> + The initial interval in which the Spark application master eagerly heartbeats to the YARN ResourceManager + when there are pending container allocation requests. It should be no larger than + <code>spark.yarn.scheduler.heartbeat.interval-ms</code>. The allocation interval will doubled on + successive eager heartbeats if pending containers still exist, until + <code>spark.yarn.scheduler.heartbeat.interval-ms</code> is reached. </td> </tr> <tr> |