diff options
| author | nemccarthy <nathan@nemccarthy.me> | 2015-03-17 09:33:11 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-03-17 09:33:28 -0700 |
| commit | febb12308dac94af20279a31e8a6013690f42f24 (patch) | |
| tree | e9d0a1530e4b8540847b7208cc0c46986a28fcdd /core/src | |
| parent | 29e39e178e9e21a038cf1aef1d110b368b6d64f7 (diff) | |
| download | spark-febb12308dac94af20279a31e8a6013690f42f24.tar.gz spark-febb12308dac94af20279a31e8a6013690f42f24.tar.bz2 spark-febb12308dac94af20279a31e8a6013690f42f24.zip | |
[SPARK-6313] Add config option to disable file locks/fetchFile cache to ...
...support NFS mounts.
This is a work around for now with the goal to find a more permanent solution.
https://issues.apache.org/jira/browse/SPARK-6313
Author: nemccarthy <nathan@nemccarthy.me>
Closes #5036 from nemccarthy/master and squashes the following commits:
2eaaf42 [nemccarthy] [SPARK-6313] Update config wording doc for spark.files.useFetchCache
5de7eb4 [nemccarthy] [SPARK-6313] Add config option to disable file locks/fetchFile cache to support NFS mounts
(cherry picked from commit 4cca3917dc30ee907e6cbd6a569b6ac58af963f7)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
Diffstat (limited to 'core/src')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/util/Utils.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala b/core/src/main/scala/org/apache/spark/util/Utils.scala index 568ff3b6de..75a40f1b42 100644 --- a/core/src/main/scala/org/apache/spark/util/Utils.scala +++ b/core/src/main/scala/org/apache/spark/util/Utils.scala @@ -402,7 +402,8 @@ private[spark] object Utils extends Logging { useCache: Boolean) { val fileName = url.split("/").last val targetFile = new File(targetDir, fileName) - if (useCache) { + val fetchCacheEnabled = conf.getBoolean("spark.files.useFetchCache", defaultValue = true) + if (useCache && fetchCacheEnabled) { val cachedFileName = s"${url.hashCode}${timestamp}_cache" val lockFileName = s"${url.hashCode}${timestamp}_lock" val localDir = new File(getLocalDir(conf)) |