diff options
| author | Michael Gummelt <mgummelt@mesosphere.io> | 2016-02-10 10:53:33 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2016-02-10 10:53:33 -0800 |
| commit | 80cb963ad963e26c3a7f8388bdd4ffd5e99aad1a (patch) | |
| tree | ebd19083ee53f66618a81280c6b6667db129b179 /docs/configuration.md | |
| parent | c0b71e0b8f3c068f2f092bb118a16611b3d38d7a (diff) | |
| download | spark-80cb963ad963e26c3a7f8388bdd4ffd5e99aad1a.tar.gz spark-80cb963ad963e26c3a7f8388bdd4ffd5e99aad1a.tar.bz2 spark-80cb963ad963e26c3a7f8388bdd4ffd5e99aad1a.zip | |
[SPARK-5095][MESOS] Support launching multiple mesos executors in coarse grained mesos mode.
This is the next iteration of tnachen's previous PR: https://github.com/apache/spark/pull/4027
In that PR, we resolved with andrewor14 and pwendell to implement the Mesos scheduler's support of `spark.executor.cores` to be consistent with YARN and Standalone. This PR implements that resolution.
This PR implements two high-level features. These two features are co-dependent, so they're implemented both here:
- Mesos support for spark.executor.cores
- Multiple executors per slave
We at Mesosphere have been working with Typesafe on a Spark/Mesos integration test suite: https://github.com/typesafehub/mesos-spark-integration-tests, which passes for this PR.
The contribution is my original work and I license the work to the project under the project's open source license.
Author: Michael Gummelt <mgummelt@mesosphere.io>
Closes #10993 from mgummelt/executor_sizing.
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 15 |
1 files changed, 10 insertions, 5 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index cd9dc1bcfc..b07c69cd4c 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -825,13 +825,18 @@ Apart from these, the following properties are also available, and may be useful </tr> <tr> <td><code>spark.executor.cores</code></td> - <td>1 in YARN mode, all the available cores on the worker in standalone mode.</td> <td> - The number of cores to use on each executor. For YARN and standalone mode only. + 1 in YARN mode, all the available cores on the worker in + standalone and Mesos coarse-grained modes. + </td> + <td> + The number of cores to use on each executor. - In standalone mode, setting this parameter allows an application to run multiple executors on - the same worker, provided that there are enough cores on that worker. Otherwise, only one - executor per application will run on each worker. + In standalone and Mesos coarse-grained modes, setting this + parameter allows an application to run multiple executors on the + same worker, provided that there are enough cores on that + worker. Otherwise, only one executor per application will run on + each worker. </td> </tr> <tr> |