diff options
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 49 |
1 files changed, 45 insertions, 4 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 567aba07f0..6717757781 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -77,12 +77,14 @@ there are at least five properties that you will commonly want to control: </tr> <tr> <td>spark.cores.max</td> - <td>(infinite)</td> + <td>(not set)</td> <td> When running on a <a href="spark-standalone.html">standalone deploy cluster</a> or a <a href="running-on-mesos.html#mesos-run-modes">Mesos cluster in "coarse-grained" - sharing mode</a>, how many CPU cores to request at most. The default will use all available cores - offered by the cluster manager. + sharing mode</a>, the maximum amount of CPU cores to request for the application from + across the cluster (not from each machine). If not set, the default will be + <code>spark.deploy.defaultCores</code> on Spark's standalone cluster manager, or + infinite (all available cores) on Mesos. </td> </tr> </table> @@ -128,7 +130,7 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> - <td>spark.ui.retained_stages</td> + <td>spark.ui.retainedStages</td> <td>1000</td> <td> How many stages the Spark UI remembers before garbage collecting. @@ -360,6 +362,14 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td>akka.x.y....</td> + <td>value</td> + <td> + An arbitrary akka configuration can be set directly on spark conf and it is applied for all the ActorSystems created spark wide for that SparkContext and its assigned executors as well. + </td> +</tr> + +<tr> <td>spark.shuffle.consolidateFiles</td> <td>false</td> <td> @@ -394,6 +404,37 @@ Apart from these, the following properties are also available, and may be useful How many times slower a task is than the median to be considered for speculation. </td> </tr> +<tr> + <td>spark.logConf</td> + <td>false</td> + <td> + Log the supplied SparkConf as INFO at start of spark context. + </td> +</tr> +<tr> + <td>spark.deploy.spreadOut</td> + <td>true</td> + <td> + Whether the standalone cluster manager should spread applications out across nodes or try + to consolidate them onto as few nodes as possible. Spreading out is usually better for + data locality in HDFS, but consolidating is more efficient for compute-intensive workloads. <br/> + <b>Note:</b> this setting needs to be configured in the standalone cluster master, not in individual + applications; you can set it through <code>SPARK_JAVA_OPTS</code> in <code>spark-env.sh</code>. + </td> +</tr> +<tr> + <td>spark.deploy.defaultCores</td> + <td>(infinite)</td> + <td> + Default number of cores to give to applications in Spark's standalone mode if they don't + set <code>spark.cores.max</code>. If not set, applications always get all available + cores unless they configure <code>spark.cores.max</code> themselves. + Set this lower on a shared cluster to prevent users from grabbing + the whole cluster by default. <br/> + <b>Note:</b> this setting needs to be configured in the standalone cluster master, not in individual + applications; you can set it through <code>SPARK_JAVA_OPTS</code> in <code>spark-env.sh</code>. + </td> +</tr> </table> ## Viewing Spark Properties |