diff options
| author | Patrick Wendell <pwendell@gmail.com> | 2013-12-04 18:39:34 -0800 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2013-12-04 18:39:34 -0800 |
| commit | b1c6fa1584099b3a1e0615c100f10ea90b1ad2c9 (patch) | |
| tree | d270cead92f1322556995aa89dfd6fabaa3e616a /docs/configuration.md | |
| parent | 380b90b9b360db9cb6a4edc1312704afe11eb31d (diff) | |
| download | spark-b1c6fa1584099b3a1e0615c100f10ea90b1ad2c9.tar.gz spark-b1c6fa1584099b3a1e0615c100f10ea90b1ad2c9.tar.bz2 spark-b1c6fa1584099b3a1e0615c100f10ea90b1ad2c9.zip | |
Document missing configs and set shuffle consolidation to false.
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 37 |
1 files changed, 36 insertions, 1 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 97183bafdb..1a3eef345c 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -327,7 +327,42 @@ Apart from these, the following properties are also available, and may be useful Too large a value decreases parallelism during broadcast (makes it slower); however, if it is too small, <code>BlockManager</code> might take a performance hit. </td> </tr> - +<tr> + <td>spark.shuffle.consolidateFiles</td> + <td>false</td> + <td> + If set to "true", consolidates intermediate files created during a shuffle. + </td> +</tr> +<tr> +<tr> + <td>spark.speculation</td> + <td>false</td> + <td> + If set to "true", performs speculative execution of tasks. This means if one or more tasks are running slowly in a stage, they will be re-launched. + </td> +</tr> +<tr> + <td>spark.speculation.interval</td> + <td>100</td> + <td> + How often Spark will check for tasks to speculate, in seconds. + </td> +</tr> +<tr> + <td>spark.speculation.quantile</td> + <td>0.75</td> + <td> + Percentage of tasks which must be complete before speculation is enabled for a particular stage. + </td> +</tr> +<tr> + <td>spark.speculation.multiplier</td> + <td>1.5</td> + <td> + How many times slower a task is than the median to be considered for speculation. + </td> +</tr> </table> # Environment Variables |