diff options
author | CodingCat <zhunansjtu@gmail.com> | 2014-06-05 11:39:35 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-06-05 11:39:35 -0700 |
commit | 89cdbb087cb2f0d03be2dd77440300c6bd61c792 (patch) | |
tree | 9fa59d1f7cfe172dbfac89acc4bc6c14903e6d8b /docs | |
parent | 7c160293d6d708718d566e700cfb407a31280b89 (diff) | |
download | spark-89cdbb087cb2f0d03be2dd77440300c6bd61c792.tar.gz spark-89cdbb087cb2f0d03be2dd77440300c6bd61c792.tar.bz2 spark-89cdbb087cb2f0d03be2dd77440300c6bd61c792.zip |
SPARK-1677: allow user to disable output dir existence checking
https://issues.apache.org/jira/browse/SPARK-1677
For compatibility with older versions of Spark it would be nice to have an option `spark.hadoop.validateOutputSpecs` (default true) for the user to disable the output directory existence checking
Author: CodingCat <zhunansjtu@gmail.com>
Closes #947 from CodingCat/SPARK-1677 and squashes the following commits:
7930f83 [CodingCat] miao
c0c0e03 [CodingCat] bug fix and doc update
5318562 [CodingCat] bug fix
13219b5 [CodingCat] allow user to disable output dir existence checking
Diffstat (limited to 'docs')
-rw-r--r-- | docs/configuration.md | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 0697f7fc2f..71fafa5734 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -487,6 +487,14 @@ Apart from these, the following properties are also available, and may be useful this duration will be cleared as well. </td> </tr> +<tr> + <td>spark.hadoop.validateOutputSpecs</td> + <td>true</td> + <td>If set to true, validates the output specification (e.g. checking if the output directory already exists) + used in saveAsHadoopFile and other variants. This can be disabled to silence exceptions due to pre-existing + output directories. We recommend that users do not disable this except if trying to achieve compatibility with + previous versions of Spark. Simply use Hadoop's FileSystem API to delete output directories by hand.</td> +</tr> </table> #### Networking |