diff options
| author | nemccarthy <nathan@nemccarthy.me> | 2015-03-17 09:33:11 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-03-17 09:33:45 -0700 |
| commit | a2a94a154bdd00753b8d5e344d712664c7151050 (patch) | |
| tree | 99a14e8367fdfe477a51e3ceef1f02fb7460dfc0 /docs | |
| parent | 9ebd6f12e67cd5995896d5bedf4a205b602109a5 (diff) | |
| download | spark-a2a94a154bdd00753b8d5e344d712664c7151050.tar.gz spark-a2a94a154bdd00753b8d5e344d712664c7151050.tar.bz2 spark-a2a94a154bdd00753b8d5e344d712664c7151050.zip | |
[SPARK-6313] Add config option to disable file locks/fetchFile cache to ...
...support NFS mounts.
This is a work around for now with the goal to find a more permanent solution.
https://issues.apache.org/jira/browse/SPARK-6313
Author: nemccarthy <nathan@nemccarthy.me>
Closes #5036 from nemccarthy/master and squashes the following commits:
2eaaf42 [nemccarthy] [SPARK-6313] Update config wording doc for spark.files.useFetchCache
5de7eb4 [nemccarthy] [SPARK-6313] Add config option to disable file locks/fetchFile cache to support NFS mounts
(cherry picked from commit 4cca3917dc30ee907e6cbd6a569b6ac58af963f7)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configuration.md | 12 |
1 files changed, 12 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index f8549dfaeb..d00c891787 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -655,6 +655,18 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.files.useFetchCache</code></td> + <td>true</td> + <td> + If set to true (default), file fetching will use a local cache that is shared by executors + that belong to the same application, which can improve task launching performance when + running many executors on the same host. If set to false, these caching optimizations will + be disabled and all executors will fetch their own copies of files. This optimization may be + disabled in order to use Spark local directories that reside on NFS filesystems (see + <a href="https://issues.apache.org/jira/browse/SPARK-6313">SPARK-6313</a> for more details). + </td> +</tr> +<tr> <td><code>spark.files.overwrite</code></td> <td>false</td> <td> |