aboutsummaryrefslogtreecommitdiff
path: root/docs/running-on-mesos.md
diff options
context:
space:
mode:
authorBrennon York <brennon.york@capitalone.com>2015-02-25 16:12:56 -0800
committerReynold Xin <rxin@databricks.com>2015-02-25 16:12:56 -0800
commit46a044a36a2aff1306f7f677e952ce253ddbefac (patch)
tree96c638352e86ac3f399afb2fe342a3e625f30e2a /docs/running-on-mesos.md
parent41e2e5acb749c25641f1f8dea5a2e1d8af319486 (diff)
downloadspark-46a044a36a2aff1306f7f677e952ce253ddbefac.tar.gz
spark-46a044a36a2aff1306f7f677e952ce253ddbefac.tar.bz2
spark-46a044a36a2aff1306f7f677e952ce253ddbefac.zip
[SPARK-1182][Docs] Sort the configuration parameters in configuration.md
Sorts all configuration options present on the `configuration.md` page to ease readability. Author: Brennon York <brennon.york@capitalone.com> Closes #3863 from brennonyork/SPARK-1182 and squashes the following commits: 5696f21 [Brennon York] fixed merge conflict with port comments 81a7b10 [Brennon York] capitalized A in Allocation e240486 [Brennon York] moved all spark.mesos properties into the running-on-mesos doc 7de5f75 [Brennon York] moved serialization from application to compression and serialization section a16fec0 [Brennon York] moved shuffle settings from network to shuffle f8fa286 [Brennon York] sorted encryption category 1023f15 [Brennon York] moved initialExecutors e9d62aa [Brennon York] fixed akka.heartbeat.interval 25e6f6f [Brennon York] moved spark.executer.user* 4625ade [Brennon York] added spark.executor.extra* items 4ee5648 [Brennon York] fixed merge conflicts 1b49234 [Brennon York] sorting mishap 2b5758b [Brennon York] sorting mishap 6fbdf42 [Brennon York] sorting mishap 55dc6f8 [Brennon York] sorted security ec34294 [Brennon York] sorted dynamic allocation 2a7c4a3 [Brennon York] sorted scheduling aa9acdc [Brennon York] sorted networking a4380b8 [Brennon York] sorted execution behavior 27f3919 [Brennon York] sorted compression and serialization 80a5bbb [Brennon York] sorted spark ui 3f32e5b [Brennon York] sorted shuffle behavior 6c51b38 [Brennon York] sorted runtime environment efe9d6f [Brennon York] sorted application properties
Diffstat (limited to 'docs/running-on-mesos.md')
-rw-r--r--docs/running-on-mesos.md24
1 files changed, 16 insertions, 8 deletions
diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md
index 78358499fd..db1173a06b 100644
--- a/docs/running-on-mesos.md
+++ b/docs/running-on-mesos.md
@@ -197,7 +197,11 @@ See the [configuration page](configuration.html) for information on Spark config
<td><code>spark.mesos.coarse</code></td>
<td>false</td>
<td>
- Set the run mode for Spark on Mesos. For more information about the run mode, refer to #Mesos Run Mode section above.
+ If set to "true", runs over Mesos clusters in
+ <a href="running-on-mesos.html#mesos-run-modes">"coarse-grained" sharing mode</a>,
+ where Spark acquires one long-lived Mesos task on each machine instead of one Mesos task per
+ Spark task. This gives lower-latency scheduling for short queries, but leaves resources in use
+ for the whole duration of the Spark job.
</td>
</tr>
<tr>
@@ -211,19 +215,23 @@ See the [configuration page](configuration.html) for information on Spark config
</tr>
<tr>
<td><code>spark.mesos.executor.home</code></td>
- <td>SPARK_HOME</td>
+ <td>driver side <code>SPARK_HOME</code></td>
<td>
- The location where the mesos executor will look for Spark binaries to execute, and uses the SPARK_HOME setting on default.
- This variable is only used when no spark.executor.uri is provided, and assumes Spark is installed on the specified location
- on each slave.
+ Set the directory in which Spark is installed on the executors in Mesos. By default, the
+ executors will simply use the driver's Spark home directory, which may not be visible to
+ them. Note that this is only relevant if a Spark binary package is not specified through
+ <code>spark.executor.uri</code>.
</td>
</tr>
<tr>
<td><code>spark.mesos.executor.memoryOverhead</code></td>
- <td>384</td>
+ <td>executor memory * 0.07, with minimum of 384</td>
<td>
- The amount of memory that Mesos executor will request for the task to account for the overhead of running the executor itself.
- The final total amount of memory allocated is the maximum value between executor memory plus memoryOverhead, and overhead fraction (1.07) plus the executor memory.
+ This value is an additive for <code>spark.executor.memory</code>, specified in MiB,
+ which is used to calculate the total Mesos task memory. A value of <code>384</code>
+ implies a 384MiB overhead. Additionally, there is a hard-coded 7% minimum
+ overhead. The final overhead will be the larger of either
+ `spark.mesos.executor.memoryOverhead` or 7% of `spark.executor.memory`.
</td>
</tr>
</table>