diff options
| author | jerryshao <sshao@hortonworks.com> | 2016-04-28 12:38:19 -0500 |
|---|---|---|
| committer | Tom Graves <tgraves@yahoo-inc.com> | 2016-04-28 12:38:19 -0500 |
| commit | 8b44bd52fa40c0fc7d34798c3654e31533fd3008 (patch) | |
| tree | 3642569112ccf117f1c91b363d2aa0445dd2e990 /docs | |
| parent | 9e785079b6ed4ea691c3c14c762a7f73fb6254bf (diff) | |
| download | spark-8b44bd52fa40c0fc7d34798c3654e31533fd3008.tar.gz spark-8b44bd52fa40c0fc7d34798c3654e31533fd3008.tar.bz2 spark-8b44bd52fa40c0fc7d34798c3654e31533fd3008.zip | |
[SPARK-6735][YARN] Add window based executor failure tracking mechanism for long running service
This work is based on twinkle-sachdeva 's proposal. In parallel to such mechanism for AM failures, here add similar mechanism for executor failure tracking, this is useful for long running Spark service to mitigate the executor failure problems.
Please help to review, tgravescs sryza and vanzin
Author: jerryshao <sshao@hortonworks.com>
Closes #10241 from jerryshao/SPARK-6735.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/running-on-yarn.md | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 09701abdb0..3bd16bf60c 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -372,6 +372,14 @@ If you need a reference to the proper location to put log files in the YARN so t </td> </tr> <tr> + <td><code>spark.yarn.executor.failuresValidityInterval</code></td> + <td>(none)</td> + <td> + Defines the validity interval for executor failure tracking. + Executor failures which are older than the validity interval will be ignored. + </td> +</tr> +<tr> <td><code>spark.yarn.submit.waitAppCompletion</code></td> <td><code>true</code></td> <td> |