diff options
| author | Sean Owen <sowen@cloudera.com> | 2016-10-26 14:23:11 +0200 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-10-26 14:23:11 +0200 |
| commit | 297813647508480d7b4b5bccd02b93b8b914301f (patch) | |
| tree | ce96991a9eb0ce5b72e96b5dfa390a3b8de67d3d /yarn/src | |
| parent | 6c7d094ec4d45a05c1ec8a418e507e45f5a88b7d (diff) | |
| download | spark-297813647508480d7b4b5bccd02b93b8b914301f.tar.gz spark-297813647508480d7b4b5bccd02b93b8b914301f.tar.bz2 spark-297813647508480d7b4b5bccd02b93b8b914301f.zip | |
[SPARK-18027][YARN] .sparkStaging not clean on RM ApplicationNotFoundException
## What changes were proposed in this pull request?
Cleanup YARN staging dir on all `KILLED`/`FAILED` paths in `monitorApplication`
## How was this patch tested?
Existing tests
Author: Sean Owen <sowen@cloudera.com>
Closes #15598 from srowen/SPARK-18027.
Diffstat (limited to 'yarn/src')
| -rw-r--r-- | yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala index 6e4f68c74c..55e4a833b6 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala @@ -1059,9 +1059,11 @@ private[spark] class Client( } catch { case e: ApplicationNotFoundException => logError(s"Application $appId not found.") + cleanupStagingDir(appId) return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED) case NonFatal(e) => logError(s"Failed to contact YARN for application $appId.", e) + // Don't necessarily clean up staging dir because status is unknown return (YarnApplicationState.FAILED, FinalApplicationStatus.FAILED) } val state = report.getYarnApplicationState |