diff options
| author | Prashant Sharma <prashant.s@imaginea.com> | 2013-12-11 10:21:53 +0530 |
|---|---|---|
| committer | Prashant Sharma <prashant.s@imaginea.com> | 2013-12-11 10:21:53 +0530 |
| commit | 603af51bb5257744ce0db28e7f10db6a2ba899ec (patch) | |
| tree | 31ad611d8d56147ec90108d2b3a92d38eae4d150 /docs/configuration.md | |
| parent | 17db6a9041d5e83d7b6fe47f9c36758d0613fcd6 (diff) | |
| parent | d2efe13574090e93c600adeacc7f6356bc196e6c (diff) | |
| download | spark-603af51bb5257744ce0db28e7f10db6a2ba899ec.tar.gz spark-603af51bb5257744ce0db28e7f10db6a2ba899ec.tar.bz2 spark-603af51bb5257744ce0db28e7f10db6a2ba899ec.zip | |
Merge branch 'master' into akka-bug-fix
Conflicts:
core/pom.xml
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
pom.xml
project/SparkBuild.scala
streaming/pom.xml
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 36 |
1 files changed, 35 insertions, 1 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 4d1a987f64..8cca77e5c0 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -348,7 +348,41 @@ Apart from these, the following properties are also available, and may be useful Too large a value decreases parallelism during broadcast (makes it slower); however, if it is too small, <code>BlockManager</code> might take a performance hit. </td> </tr> - +<tr> + <td>spark.shuffle.consolidateFiles</td> + <td>false</td> + <td> + If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance for shuffles with large numbers of reduce tasks. It is recommended to set this to "true" when using ext4 or xfs filesystems. On ext3, this option might degrade performance on machines with many (>8) cores due to filesystem limitations. + </td> +</tr> +<tr> + <td>spark.speculation</td> + <td>false</td> + <td> + If set to "true", performs speculative execution of tasks. This means if one or more tasks are running slowly in a stage, they will be re-launched. + </td> +</tr> +<tr> + <td>spark.speculation.interval</td> + <td>100</td> + <td> + How often Spark will check for tasks to speculate, in milliseconds. + </td> +</tr> +<tr> + <td>spark.speculation.quantile</td> + <td>0.75</td> + <td> + Percentage of tasks which must be complete before speculation is enabled for a particular stage. + </td> +</tr> +<tr> + <td>spark.speculation.multiplier</td> + <td>1.5</td> + <td> + How many times slower a task is than the median to be considered for speculation. + </td> +</tr> </table> # Environment Variables |