From 978cd5f125eb5a410bad2e60bf8385b11cf1b978 Mon Sep 17 00:00:00 2001 From: Philipp Hoffmann Date: Mon, 25 Jul 2016 20:14:47 +0100 Subject: [SPARK-15271][MESOS] Allow force pulling executor docker images ## What changes were proposed in this pull request? Mesos agents by default will not pull docker images which are cached locally already. In order to run Spark executors from mutable tags like `:latest` this commit introduces a Spark setting `spark.mesos.executor.docker.forcePullImage`. Setting this flag to true will tell the Mesos agent to force pull the docker image (default is `false` which is consistent with the previous implementation and Mesos' default behaviour). ## How was this patch tested? I ran a sample application including this change on a Mesos cluster and verified the correct behaviour for both, with and without, force pulling the executor image. As expected the image is being force pulled if the flag is set. Author: Philipp Hoffmann Closes #13051 from philipphoffmann/force-pull-image. --- docs/_config.yml | 2 +- docs/running-on-mesos.md | 12 ++++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) (limited to 'docs') diff --git a/docs/_config.yml b/docs/_config.yml index be3d8a2fe6..bbb576e0e7 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -18,6 +18,6 @@ SPARK_VERSION: 2.1.0-SNAPSHOT SPARK_VERSION_SHORT: 2.1.0 SCALA_BINARY_VERSION: "2.11" SCALA_VERSION: "2.11.7" -MESOS_VERSION: 0.21.0 +MESOS_VERSION: 0.22.0 SPARK_ISSUE_TRACKER_URL: https://issues.apache.org/jira/browse/SPARK SPARK_GITHUB_URL: https://github.com/apache/spark diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md index 10dc9ce890..ce888b5445 100644 --- a/docs/running-on-mesos.md +++ b/docs/running-on-mesos.md @@ -260,6 +260,10 @@ have Mesos download Spark via the usual methods. Requires Mesos version 0.20.1 or later. +Note that by default Mesos agents will not pull the image if it already exists on the agent. If you use mutable image +tags you can set `spark.mesos.executor.docker.forcePullImage` to `true` in order to force the agent to always pull the +image before running the executor. Force pulling images is only available in Mesos version 0.22 and above. + # Running Alongside Hadoop You can run Spark and Mesos alongside your existing Hadoop cluster by just launching them as a @@ -334,6 +338,14 @@ See the [configuration page](configuration.html) for information on Spark config the installed path of the Mesos library can be specified with spark.executorEnv.MESOS_NATIVE_JAVA_LIBRARY. + + spark.mesos.executor.docker.forcePullImage + false + + Force Mesos agents to pull the image specified in spark.mesos.executor.docker.image. + By default Mesos agents will not pull images they already have cached. + + spark.mesos.executor.docker.volumes (none) -- cgit v1.2.3