diff options
| author | Philipp Hoffmann <mail@philipphoffmann.de> | 2016-07-26 16:09:10 +0100 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-07-26 16:09:10 +0100 |
| commit | 0869b3a5f028b64c2da511e70b02ab42f65fc949 (patch) | |
| tree | d924a19e06814f6fb1ae5a2028c8cddbdba4980a /docs | |
| parent | a2abb583caaec9a2cecd5d65b05d172fc096c125 (diff) | |
| download | spark-0869b3a5f028b64c2da511e70b02ab42f65fc949.tar.gz spark-0869b3a5f028b64c2da511e70b02ab42f65fc949.tar.bz2 spark-0869b3a5f028b64c2da511e70b02ab42f65fc949.zip | |
[SPARK-15271][MESOS] Allow force pulling executor docker images
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Mesos agents by default will not pull docker images which are cached
locally already. In order to run Spark executors from mutable tags like
`:latest` this commit introduces a Spark setting
(`spark.mesos.executor.docker.forcePullImage`). Setting this flag to
true will tell the Mesos agent to force pull the docker image (default is `false` which is consistent with the previous
implementation and Mesos' default
behaviour).
Author: Philipp Hoffmann <mail@philipphoffmann.de>
Closes #14348 from philipphoffmann/force-pull-image.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/_config.yml | 2 | ||||
| -rw-r--r-- | docs/running-on-mesos.md | 12 |
2 files changed, 13 insertions, 1 deletions
diff --git a/docs/_config.yml b/docs/_config.yml index be3d8a2fe6..bbb576e0e7 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -18,6 +18,6 @@ SPARK_VERSION: 2.1.0-SNAPSHOT SPARK_VERSION_SHORT: 2.1.0 SCALA_BINARY_VERSION: "2.11" SCALA_VERSION: "2.11.7" -MESOS_VERSION: 0.21.0 +MESOS_VERSION: 0.22.0 SPARK_ISSUE_TRACKER_URL: https://issues.apache.org/jira/browse/SPARK SPARK_GITHUB_URL: https://github.com/apache/spark diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md index 10dc9ce890..ce888b5445 100644 --- a/docs/running-on-mesos.md +++ b/docs/running-on-mesos.md @@ -260,6 +260,10 @@ have Mesos download Spark via the usual methods. Requires Mesos version 0.20.1 or later. +Note that by default Mesos agents will not pull the image if it already exists on the agent. If you use mutable image +tags you can set `spark.mesos.executor.docker.forcePullImage` to `true` in order to force the agent to always pull the +image before running the executor. Force pulling images is only available in Mesos version 0.22 and above. + # Running Alongside Hadoop You can run Spark and Mesos alongside your existing Hadoop cluster by just launching them as a @@ -335,6 +339,14 @@ See the [configuration page](configuration.html) for information on Spark config </td> </tr> <tr> + <td><code>spark.mesos.executor.docker.forcePullImage</code></td> + <td>false</td> + <td> + Force Mesos agents to pull the image specified in <code>spark.mesos.executor.docker.image</code>. + By default Mesos agents will not pull images they already have cached. + </td> +</tr> +<tr> <td><code>spark.mesos.executor.docker.volumes</code></td> <td>(none)</td> <td> |