diff options
| author | Gera Shegalov <gera@twitter.com> | 2014-07-22 21:05:12 -0500 |
|---|---|---|
| committer | Thomas Graves <tgraves@apache.org> | 2014-07-22 21:05:12 -0500 |
| commit | 02e45729472a22a31629cdd17dc3836ba6810189 (patch) | |
| tree | 106d7346267aba0e7045cd6c0d662a6b99388c31 /yarn | |
| parent | ddadf1b00470b9d7bf7386dacf198d41407a0a2b (diff) | |
| download | spark-02e45729472a22a31629cdd17dc3836ba6810189.tar.gz spark-02e45729472a22a31629cdd17dc3836ba6810189.tar.bz2 spark-02e45729472a22a31629cdd17dc3836ba6810189.zip | |
[YARN] SPARK-2577: File upload to viewfs is broken due to mount point re...
Opting to the option 2 defined in SPARK-2577, i.e., retrieve and pass the correct file system object to addResource.
Author: Gera Shegalov <gera@twitter.com>
Closes #1483 from gerashegalov/master and squashes the following commits:
90c9087 [Gera Shegalov] [YARN] SPARK-2577: File upload to viewfs is broken due to mount point resolution
Diffstat (limited to 'yarn')
| -rw-r--r-- | yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala index 556f493429..a1298e8f30 100644 --- a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala +++ b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala @@ -232,7 +232,8 @@ trait ClientBase extends Logging { if (!ClientBase.LOCAL_SCHEME.equals(localURI.getScheme())) { val setPermissions = if (destName.equals(ClientBase.APP_JAR)) true else false val destPath = copyRemoteFile(dst, qualifyForLocal(localURI), replication, setPermissions) - distCacheMgr.addResource(fs, conf, destPath, localResources, LocalResourceType.FILE, + val destFs = FileSystem.get(destPath.toUri(), conf) + distCacheMgr.addResource(destFs, conf, destPath, localResources, LocalResourceType.FILE, destName, statCache) } else if (confKey != null) { sparkConf.set(confKey, localPath) |