diff options
author | Marcelo Vanzin <vanzin@cloudera.com> | 2016-06-21 12:48:06 -0500 |
---|---|---|
committer | Tom Graves <tgraves@yahoo-inc.com> | 2016-06-21 12:48:06 -0500 |
commit | bcb0258ae62f23f71a067c1304232f272d7374aa (patch) | |
tree | 6ed9dd60a2bea917cf8d1fdd161702d36058465c /yarn/src/main | |
parent | 93338807aafdb2db9fb036ceadee1467cd367cdd (diff) | |
download | spark-bcb0258ae62f23f71a067c1304232f272d7374aa.tar.gz spark-bcb0258ae62f23f71a067c1304232f272d7374aa.tar.bz2 spark-bcb0258ae62f23f71a067c1304232f272d7374aa.zip |
[SPARK-16080][YARN] Set correct link name for conf archive in executors.
This makes sure the files are in the executor's classpath as they're
expected to be. Also update the unit test to make sure the files are
there as expected.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #13792 from vanzin/SPARK-16080.
Diffstat (limited to 'yarn/src/main')
-rw-r--r-- | yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala | 14 |
1 files changed, 10 insertions, 4 deletions
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala index 4df90d7b6b..847d1de50f 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala @@ -160,11 +160,17 @@ private[spark] class ApplicationMaster( } // Distribute the conf archive to executors. - sparkConf.get(CACHED_CONF_ARCHIVE).foreach { uri => - val fs = FileSystem.get(new URI(uri), yarnConf) + sparkConf.get(CACHED_CONF_ARCHIVE).foreach { path => + val uri = new URI(path) + val fs = FileSystem.get(uri, yarnConf) val status = fs.getFileStatus(new Path(uri)) - setupDistributedCache(uri, LocalResourceType.ARCHIVE, status.getModificationTime().toString, - status.getLen.toString, LocalResourceVisibility.PRIVATE.name()) + // SPARK-16080: Make sure to use the correct name for the destination when distributing the + // conf archive to executors. + val destUri = new URI(uri.getScheme(), uri.getRawSchemeSpecificPart(), + Client.LOCALIZED_CONF_DIR) + setupDistributedCache(destUri.toString(), LocalResourceType.ARCHIVE, + status.getModificationTime().toString, status.getLen.toString, + LocalResourceVisibility.PRIVATE.name()) } // Clean up the configuration so it doesn't show up in the Web UI (since it's really noisy). |