diff options
author | Reynold Xin <rxin@apache.org> | 2014-01-27 17:08:35 -0800 |
---|---|---|
committer | Reynold Xin <rxin@apache.org> | 2014-01-27 17:08:35 -0800 |
commit | 84670f2715392859624df290c1b52eb4ed4a9cb1 (patch) | |
tree | 5d3a95afe7e9e7527a271886a4b4dd1f44820434 /docs | |
parent | 3d5c03e2305777b8a32f2e196e3b73ab221b3e79 (diff) | |
parent | 584323c6b13e0d4624eb39360d0caff6c8232aac (diff) | |
download | spark-84670f2715392859624df290c1b52eb4ed4a9cb1.tar.gz spark-84670f2715392859624df290c1b52eb4ed4a9cb1.tar.bz2 spark-84670f2715392859624df290c1b52eb4ed4a9cb1.zip |
Merge pull request #466 from liyinan926/file-overwrite-new
Allow files added through SparkContext.addFile() to be overwritten
This is useful for the cases when a file needs to be refreshed and downloaded by the executors periodically. For example, a possible use case is: the driver periodically renews a Hadoop delegation token and writes it to a token file. The token file needs to be downloaded by the executors whenever it gets renewed. However, the current implementation throws an exception when the target file exists and its contents do not match those of the new source. This PR adds an option to allow files to be overwritten to support use cases similar to the above.
Diffstat (limited to 'docs')
-rw-r--r-- | docs/configuration.md | 7 |
1 files changed, 7 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 3bb655075f..5c4714dc24 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -454,6 +454,13 @@ Apart from these, the following properties are also available, and may be useful the whole cluster by default. <br/> <b>Note:</b> this setting needs to be configured in the standalone cluster master, not in individual applications; you can set it through <code>SPARK_JAVA_OPTS</code> in <code>spark-env.sh</code>. +</td> +</tr> +<tr> + <td>spark.files.overwrite</td> + <td>false</td> + <td> + Whether to overwrite files added through SparkContext.addFile() when the target file exists and its contents do not match those of the source. </td> </tr> </table> |