From 0869b3a5f028b64c2da511e70b02ab42f65fc949 Mon Sep 17 00:00:00 2001 From: Philipp Hoffmann Date: Tue, 26 Jul 2016 16:09:10 +0100 Subject: [SPARK-15271][MESOS] Allow force pulling executor docker images ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Mesos agents by default will not pull docker images which are cached locally already. In order to run Spark executors from mutable tags like `:latest` this commit introduces a Spark setting (`spark.mesos.executor.docker.forcePullImage`). Setting this flag to true will tell the Mesos agent to force pull the docker image (default is `false` which is consistent with the previous implementation and Mesos' default behaviour). Author: Philipp Hoffmann Closes #14348 from philipphoffmann/force-pull-image. --- docs/running-on-mesos.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) (limited to 'docs/running-on-mesos.md') diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md index 10dc9ce890..ce888b5445 100644 --- a/docs/running-on-mesos.md +++ b/docs/running-on-mesos.md @@ -260,6 +260,10 @@ have Mesos download Spark via the usual methods. Requires Mesos version 0.20.1 or later. +Note that by default Mesos agents will not pull the image if it already exists on the agent. If you use mutable image +tags you can set `spark.mesos.executor.docker.forcePullImage` to `true` in order to force the agent to always pull the +image before running the executor. Force pulling images is only available in Mesos version 0.22 and above. + # Running Alongside Hadoop You can run Spark and Mesos alongside your existing Hadoop cluster by just launching them as a @@ -334,6 +338,14 @@ See the [configuration page](configuration.html) for information on Spark config the installed path of the Mesos library can be specified with spark.executorEnv.MESOS_NATIVE_JAVA_LIBRARY. + + spark.mesos.executor.docker.forcePullImage + false + + Force Mesos agents to pull the image specified in spark.mesos.executor.docker.image. + By default Mesos agents will not pull images they already have cached. + + spark.mesos.executor.docker.volumes (none) -- cgit v1.2.3