diff options
| author | huangzhaowei <carlmartinmax@gmail.com> | 2016-02-25 09:14:19 -0600 |
|---|---|---|
| committer | Tom Graves <tgraves@yahoo-inc.com> | 2016-02-25 09:14:19 -0600 |
| commit | 5fcf4c2bfce4b7e3543815c8e49ffdec8072c9a2 (patch) | |
| tree | b66776d9af6e3cc09c09ac85b602c689457c7103 /yarn/src | |
| parent | 157fe64f3ecbd13b7286560286e50235eecfe30e (diff) | |
| download | spark-5fcf4c2bfce4b7e3543815c8e49ffdec8072c9a2.tar.gz spark-5fcf4c2bfce4b7e3543815c8e49ffdec8072c9a2.tar.bz2 spark-5fcf4c2bfce4b7e3543815c8e49ffdec8072c9a2.zip | |
[SPARK-12316] Wait a minutes to avoid cycle calling.
When application end, AM will clean the staging dir.
But if the driver trigger to update the delegation token, it will can't find the right token file and then it will endless cycle call the method 'updateCredentialsIfRequired'.
Then it lead driver StackOverflowError.
https://issues.apache.org/jira/browse/SPARK-12316
Author: huangzhaowei <carlmartinmax@gmail.com>
Closes #10475 from SaintBacchus/SPARK-12316.
Diffstat (limited to 'yarn/src')
| -rw-r--r-- | yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala | 5 |
1 files changed, 4 insertions, 1 deletions
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala index 9d99c0d93f..6474acc3dc 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala @@ -76,7 +76,10 @@ private[spark] class ExecutorDelegationTokenUpdater( SparkHadoopUtil.get.getTimeFromNowToRenewal( sparkConf, 0.8, UserGroupInformation.getCurrentUser.getCredentials) if (timeFromNowToRenewal <= 0) { - executorUpdaterRunnable.run() + // We just checked for new credentials but none were there, wait a minute and retry. + // This handles the shutdown case where the staging directory may have been removed(see + // SPARK-12316 for more details). + delegationTokenRenewer.schedule(executorUpdaterRunnable, 1, TimeUnit.MINUTES) } else { logInfo(s"Scheduling token refresh from HDFS in $timeFromNowToRenewal millis.") delegationTokenRenewer.schedule( |