diff options
| author | Christophe Préaud <christophe.preaud@kelkoo.com> | 2014-12-08 11:44:54 -0800 |
|---|---|---|
| committer | Josh Rosen <rosenville@gmail.com> | 2014-12-08 11:44:54 -0800 |
| commit | ab2abcb5ef925f15fa0e08d34a79b94a7b6578ef (patch) | |
| tree | 7bc29b072a92c3371af7c1c020bee2f525ea9e32 /core | |
| parent | 8817fc7fe8785d7b11138ca744f22f7e70f1f0a0 (diff) | |
| download | spark-ab2abcb5ef925f15fa0e08d34a79b94a7b6578ef.tar.gz spark-ab2abcb5ef925f15fa0e08d34a79b94a7b6578ef.tar.bz2 spark-ab2abcb5ef925f15fa0e08d34a79b94a7b6578ef.zip | |
[SPARK-4764] Ensure that files are fetched atomically
tempFile is created in the same directory than targetFile, so that the
move from tempFile to targetFile is always atomic
Author: Christophe Préaud <christophe.preaud@kelkoo.com>
Closes #2855 from preaudc/master and squashes the following commits:
9ba89ca [Christophe Préaud] Ensure that files are fetched atomically
54419ae [Christophe Préaud] Merge remote-tracking branch 'upstream/master'
c6a5590 [Christophe Préaud] Revert commit 8ea871f8130b2490f1bad7374a819bf56f0ccbbd
7456a33 [Christophe Préaud] Merge remote-tracking branch 'upstream/master'
8ea871f [Christophe Préaud] Ensure that files are fetched atomically
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/util/Utils.scala | 3 |
1 files changed, 1 insertions, 2 deletions
diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala b/core/src/main/scala/org/apache/spark/util/Utils.scala index 336b0798ca..9c04e45a58 100644 --- a/core/src/main/scala/org/apache/spark/util/Utils.scala +++ b/core/src/main/scala/org/apache/spark/util/Utils.scala @@ -425,8 +425,7 @@ private[spark] object Utils extends Logging { conf: SparkConf, securityMgr: SecurityManager, hadoopConf: Configuration) { - val tempDir = getLocalDir(conf) - val tempFile = File.createTempFile("fetchFileTemp", null, new File(tempDir)) + val tempFile = File.createTempFile("fetchFileTemp", null, new File(targetDir.getAbsolutePath)) val targetFile = new File(targetDir, filename) val uri = new URI(url) val fileOverwrite = conf.getBoolean("spark.files.overwrite", defaultValue = false) |