diff options
author | Stavros Kontopoulos <stavros.kontopoulos@lightbend.com> | 2016-08-18 12:19:19 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2016-08-18 12:19:19 +0100 |
commit | b81421afb04959bb22b53653be0a09c1f1c5845f (patch) | |
tree | b627c4379441809ee1a14e0800c209b679e63a42 | |
parent | e82dbe600e0d36d76cd5607a77c3243a26777b77 (diff) | |
download | spark-b81421afb04959bb22b53653be0a09c1f1c5845f.tar.gz spark-b81421afb04959bb22b53653be0a09c1f1c5845f.tar.bz2 spark-b81421afb04959bb22b53653be0a09c1f1c5845f.zip |
[SPARK-17087][MESOS] Documentation for Making Spark on Mesos honor port restrictions
## What changes were proposed in this pull request?
- adds documentation for https://issues.apache.org/jira/browse/SPARK-11714
## How was this patch tested?
Doc no test needed.
Author: Stavros Kontopoulos <stavros.kontopoulos@lightbend.com>
Closes #14667 from skonto/add_doc.
-rw-r--r-- | docs/running-on-mesos.md | 10 |
1 files changed, 10 insertions, 0 deletions
diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md index a6ce34c761..173961deaa 100644 --- a/docs/running-on-mesos.md +++ b/docs/running-on-mesos.md @@ -207,6 +207,16 @@ The scheduler will start executors round-robin on the offers Mesos gives it, but there are no spread guarantees, as Mesos does not provide such guarantees on the offer stream. +In this mode spark executors will honor port allocation if such is +provided from the user. Specifically if the user defines +`spark.executor.port` or `spark.blockManager.port` in Spark configuration, +the mesos scheduler will check the available offers for a valid port +range containing the port numbers. If no such range is available it will +not launch any task. If no restriction is imposed on port numbers by the +user, ephemeral ports are used as usual. This port honouring implementation +implies one task per host if the user defines a port. In the future network +isolation shall be supported. + The benefit of coarse-grained mode is much lower startup overhead, but at the cost of reserving Mesos resources for the complete duration of the application. To configure your job to dynamically adjust to its |