diff options
author | Philipp Hoffmann <mail@philipphoffmann.de> | 2016-07-26 16:09:10 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2016-07-26 16:09:10 +0100 |
commit | 0869b3a5f028b64c2da511e70b02ab42f65fc949 (patch) | |
tree | d924a19e06814f6fb1ae5a2028c8cddbdba4980a /docs/running-on-mesos.md | |
parent | a2abb583caaec9a2cecd5d65b05d172fc096c125 (diff) | |
download | spark-0869b3a5f028b64c2da511e70b02ab42f65fc949.tar.gz spark-0869b3a5f028b64c2da511e70b02ab42f65fc949.tar.bz2 spark-0869b3a5f028b64c2da511e70b02ab42f65fc949.zip |
[SPARK-15271][MESOS] Allow force pulling executor docker images
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Mesos agents by default will not pull docker images which are cached
locally already. In order to run Spark executors from mutable tags like
`:latest` this commit introduces a Spark setting
(`spark.mesos.executor.docker.forcePullImage`). Setting this flag to
true will tell the Mesos agent to force pull the docker image (default is `false` which is consistent with the previous
implementation and Mesos' default
behaviour).
Author: Philipp Hoffmann <mail@philipphoffmann.de>
Closes #14348 from philipphoffmann/force-pull-image.
Diffstat (limited to 'docs/running-on-mesos.md')
-rw-r--r-- | docs/running-on-mesos.md | 12 |
1 files changed, 12 insertions, 0 deletions
diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md index 10dc9ce890..ce888b5445 100644 --- a/docs/running-on-mesos.md +++ b/docs/running-on-mesos.md @@ -260,6 +260,10 @@ have Mesos download Spark via the usual methods. Requires Mesos version 0.20.1 or later. +Note that by default Mesos agents will not pull the image if it already exists on the agent. If you use mutable image +tags you can set `spark.mesos.executor.docker.forcePullImage` to `true` in order to force the agent to always pull the +image before running the executor. Force pulling images is only available in Mesos version 0.22 and above. + # Running Alongside Hadoop You can run Spark and Mesos alongside your existing Hadoop cluster by just launching them as a @@ -335,6 +339,14 @@ See the [configuration page](configuration.html) for information on Spark config </td> </tr> <tr> + <td><code>spark.mesos.executor.docker.forcePullImage</code></td> + <td>false</td> + <td> + Force Mesos agents to pull the image specified in <code>spark.mesos.executor.docker.image</code>. + By default Mesos agents will not pull images they already have cached. + </td> +</tr> +<tr> <td><code>spark.mesos.executor.docker.volumes</code></td> <td>(none)</td> <td> |