diff options
author | Terence Yim <terence@cask.co> | 2016-02-25 13:29:30 +0000 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2016-02-25 13:29:30 +0000 |
commit | fae88af18445c5a88212b4644e121de4b30ce027 (patch) | |
tree | f0256246dc4aac9bbc592840d80e4a6df131eb0f | |
parent | 6f8e835c68dff6fcf97326dc617132a41ff9d043 (diff) | |
download | spark-fae88af18445c5a88212b4644e121de4b30ce027.tar.gz spark-fae88af18445c5a88212b4644e121de4b30ce027.tar.bz2 spark-fae88af18445c5a88212b4644e121de4b30ce027.zip |
[SPARK-13441][YARN] Fix NPE in yarn Client.createConfArchive method
## What changes were proposed in this pull request?
Instead of using result of File.listFiles() directly, which may throw NPE, check for null first. If it is null, log a warning instead
## How was the this patch tested?
Ran the ./dev/run-tests locally
Tested manually on a cluster
Author: Terence Yim <terence@cask.co>
Closes #11337 from chtyim/fixes/SPARK-13441-null-check.
-rw-r--r-- | yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 11 |
1 files changed, 8 insertions, 3 deletions
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala index d4ca255953..530f1d753b 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala @@ -537,9 +537,14 @@ private[spark] class Client( sys.env.get(envKey).foreach { path => val dir = new File(path) if (dir.isDirectory()) { - dir.listFiles().foreach { file => - if (file.isFile && !hadoopConfFiles.contains(file.getName())) { - hadoopConfFiles(file.getName()) = file + val files = dir.listFiles() + if (files == null) { + logWarning("Failed to list files under directory " + dir) + } else { + files.foreach { file => + if (file.isFile && !hadoopConfFiles.contains(file.getName())) { + hadoopConfFiles(file.getName()) = file + } } } } |