diff options
| author | WangTaoTheTonic <barneystinson@aliyun.com> | 2015-01-09 08:10:09 -0600 |
|---|---|---|
| committer | Thomas Graves <tgraves@apache.org> | 2015-01-09 08:10:09 -0600 |
| commit | f3da4bd7289d493014ad3c5176ada60794dfcfe0 (patch) | |
| tree | 89998bb51b9b7e8efefe9518126479809722a9c9 /yarn | |
| parent | 167a5ab0bd1d37f3ac23bec49e484a238610cf75 (diff) | |
| download | spark-f3da4bd7289d493014ad3c5176ada60794dfcfe0.tar.gz spark-f3da4bd7289d493014ad3c5176ada60794dfcfe0.tar.bz2 spark-f3da4bd7289d493014ad3c5176ada60794dfcfe0.zip | |
[SPARK-5169][YARN]fetch the correct max attempts
Soryy for fetching the wrong max attempts in this commit https://github.com/apache/spark/commit/8fdd48959c93b9cf809f03549e2ae6c4687d1fcd.
We need to fix it now.
tgravescs
If we set an spark.yarn.maxAppAttempts which is larger than `yarn.resourcemanager.am.max-attempts` in yarn side, it will be overrided as described here:
>The maximum number of application attempts. It's a global setting for all application masters. Each application master can specify its individual maximum number of application attempts via the API, but the individual number cannot be more than the global upper bound. If it is, the resourcemanager will override it. The default number is set to 2, to allow at least one retry for AM.
http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
Author: WangTaoTheTonic <barneystinson@aliyun.com>
Closes #3942 from WangTaoTheTonic/HOTFIX and squashes the following commits:
9ac16ce [WangTaoTheTonic] fetch the correct max attempts
Diffstat (limited to 'yarn')
| -rw-r--r-- | yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala | 12 |
1 files changed, 9 insertions, 3 deletions
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala index e183efccbb..b45e599588 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala @@ -121,9 +121,15 @@ private[spark] class YarnRMClient(args: ApplicationMasterArguments) extends Logg /** Returns the maximum number of attempts to register the AM. */ def getMaxRegAttempts(sparkConf: SparkConf, yarnConf: YarnConfiguration): Int = { - sparkConf.getOption("spark.yarn.maxAppAttempts").map(_.toInt).getOrElse( - yarnConf.getInt( - YarnConfiguration.RM_AM_MAX_ATTEMPTS, YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)) + val sparkMaxAttempts = sparkConf.getOption("spark.yarn.maxAppAttempts").map(_.toInt) + val yarnMaxAttempts = yarnConf.getInt( + YarnConfiguration.RM_AM_MAX_ATTEMPTS, YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS) + val retval: Int = sparkMaxAttempts match { + case Some(x) => if (x <= yarnMaxAttempts) x else yarnMaxAttempts + case None => yarnMaxAttempts + } + + retval } } |