diff options
author | Brennon York <brennon.york@capitalone.com> | 2015-02-25 16:12:56 -0800 |
---|---|---|
committer | Reynold Xin <rxin@databricks.com> | 2015-02-25 16:12:56 -0800 |
commit | 46a044a36a2aff1306f7f677e952ce253ddbefac (patch) | |
tree | 96c638352e86ac3f399afb2fe342a3e625f30e2a /docs/running-on-mesos.md | |
parent | 41e2e5acb749c25641f1f8dea5a2e1d8af319486 (diff) | |
download | spark-46a044a36a2aff1306f7f677e952ce253ddbefac.tar.gz spark-46a044a36a2aff1306f7f677e952ce253ddbefac.tar.bz2 spark-46a044a36a2aff1306f7f677e952ce253ddbefac.zip |
[SPARK-1182][Docs] Sort the configuration parameters in configuration.md
Sorts all configuration options present on the `configuration.md` page to ease readability.
Author: Brennon York <brennon.york@capitalone.com>
Closes #3863 from brennonyork/SPARK-1182 and squashes the following commits:
5696f21 [Brennon York] fixed merge conflict with port comments
81a7b10 [Brennon York] capitalized A in Allocation
e240486 [Brennon York] moved all spark.mesos properties into the running-on-mesos doc
7de5f75 [Brennon York] moved serialization from application to compression and serialization section
a16fec0 [Brennon York] moved shuffle settings from network to shuffle
f8fa286 [Brennon York] sorted encryption category
1023f15 [Brennon York] moved initialExecutors
e9d62aa [Brennon York] fixed akka.heartbeat.interval
25e6f6f [Brennon York] moved spark.executer.user*
4625ade [Brennon York] added spark.executor.extra* items
4ee5648 [Brennon York] fixed merge conflicts
1b49234 [Brennon York] sorting mishap
2b5758b [Brennon York] sorting mishap
6fbdf42 [Brennon York] sorting mishap
55dc6f8 [Brennon York] sorted security
ec34294 [Brennon York] sorted dynamic allocation
2a7c4a3 [Brennon York] sorted scheduling
aa9acdc [Brennon York] sorted networking
a4380b8 [Brennon York] sorted execution behavior
27f3919 [Brennon York] sorted compression and serialization
80a5bbb [Brennon York] sorted spark ui
3f32e5b [Brennon York] sorted shuffle behavior
6c51b38 [Brennon York] sorted runtime environment
efe9d6f [Brennon York] sorted application properties
Diffstat (limited to 'docs/running-on-mesos.md')
-rw-r--r-- | docs/running-on-mesos.md | 24 |
1 files changed, 16 insertions, 8 deletions
diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md index 78358499fd..db1173a06b 100644 --- a/docs/running-on-mesos.md +++ b/docs/running-on-mesos.md @@ -197,7 +197,11 @@ See the [configuration page](configuration.html) for information on Spark config <td><code>spark.mesos.coarse</code></td> <td>false</td> <td> - Set the run mode for Spark on Mesos. For more information about the run mode, refer to #Mesos Run Mode section above. + If set to "true", runs over Mesos clusters in + <a href="running-on-mesos.html#mesos-run-modes">"coarse-grained" sharing mode</a>, + where Spark acquires one long-lived Mesos task on each machine instead of one Mesos task per + Spark task. This gives lower-latency scheduling for short queries, but leaves resources in use + for the whole duration of the Spark job. </td> </tr> <tr> @@ -211,19 +215,23 @@ See the [configuration page](configuration.html) for information on Spark config </tr> <tr> <td><code>spark.mesos.executor.home</code></td> - <td>SPARK_HOME</td> + <td>driver side <code>SPARK_HOME</code></td> <td> - The location where the mesos executor will look for Spark binaries to execute, and uses the SPARK_HOME setting on default. - This variable is only used when no spark.executor.uri is provided, and assumes Spark is installed on the specified location - on each slave. + Set the directory in which Spark is installed on the executors in Mesos. By default, the + executors will simply use the driver's Spark home directory, which may not be visible to + them. Note that this is only relevant if a Spark binary package is not specified through + <code>spark.executor.uri</code>. </td> </tr> <tr> <td><code>spark.mesos.executor.memoryOverhead</code></td> - <td>384</td> + <td>executor memory * 0.07, with minimum of 384</td> <td> - The amount of memory that Mesos executor will request for the task to account for the overhead of running the executor itself. - The final total amount of memory allocated is the maximum value between executor memory plus memoryOverhead, and overhead fraction (1.07) plus the executor memory. + This value is an additive for <code>spark.executor.memory</code>, specified in MiB, + which is used to calculate the total Mesos task memory. A value of <code>384</code> + implies a 384MiB overhead. Additionally, there is a hard-coded 7% minimum + overhead. The final overhead will be the larger of either + `spark.mesos.executor.memoryOverhead` or 7% of `spark.executor.memory`. </td> </tr> </table> |