diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2014-10-17 13:45:10 -0500 |
|---|---|---|
| committer | Thomas Graves <tgraves@apache.org> | 2014-10-17 13:45:10 -0500 |
| commit | 803e7f087797bae643754f8db88848a17282ca6e (patch) | |
| tree | 5fb0c7c89be44530477491a6ff9f0b43403ed544 /yarn/common | |
| parent | c351862064ed7d2031ea4c8bf33881e5f702ea0a (diff) | |
| download | spark-803e7f087797bae643754f8db88848a17282ca6e.tar.gz spark-803e7f087797bae643754f8db88848a17282ca6e.tar.bz2 spark-803e7f087797bae643754f8db88848a17282ca6e.zip | |
[SPARK-3979] [yarn] Use fs's default replication.
This avoids issues when HDFS is configured in a way that would not
allow the hardcoded default replication of "3".
Note: getDefaultReplication(Path) was added in 0.23.3, and the oldest
one available on Maven Central is 0.23.7, so I chose to not add code
to access that method via reflection.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #2831 from vanzin/SPARK-3979 and squashes the following commits:
b0e3a97 [Marcelo Vanzin] [SPARK-3979] [yarn] Use fs's default replication.
Diffstat (limited to 'yarn/common')
| -rw-r--r-- | yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala index 14a0386b78..0efac4ea63 100644 --- a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala +++ b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala @@ -143,7 +143,8 @@ private[spark] trait ClientBase extends Logging { val nns = getNameNodesToAccess(sparkConf) + dst obtainTokensForNamenodes(nns, hadoopConf, credentials) - val replication = sparkConf.getInt("spark.yarn.submit.file.replication", 3).toShort + val replication = sparkConf.getInt("spark.yarn.submit.file.replication", + fs.getDefaultReplication(dst)).toShort val localResources = HashMap[String, LocalResource]() FileSystem.mkdirs(fs, dst, new FsPermission(STAGING_DIR_PERMISSION)) |