From 266b92faffb66af24d8ed2725beb80770a2d91f8 Mon Sep 17 00:00:00 2001 From: Michael Gummelt Date: Fri, 29 Jul 2016 05:50:47 -0700 Subject: [SPARK-16637] Unified containerizer ## What changes were proposed in this pull request? New config var: spark.mesos.docker.containerizer={"mesos","docker" (default)} This adds support for running docker containers via the Mesos unified containerizer: http://mesos.apache.org/documentation/latest/container-image/ The benefit is losing the dependency on `dockerd`, and all the costs which it incurs. I've also updated the supported Mesos version to 0.28.2 for support of the required protobufs. This is blocked on: https://github.com/apache/spark/pull/14167 ## How was this patch tested? - manually testing jobs submitted with both "mesos" and "docker" settings for the new config var. - spark/mesos integration test suite Author: Michael Gummelt Closes #14275 from mgummelt/unified-containerizer. --- docs/_config.yml | 2 +- docs/running-on-mesos.md | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) (limited to 'docs') diff --git a/docs/_config.yml b/docs/_config.yml index bbb576e0e7..e4fc093fe7 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -18,6 +18,6 @@ SPARK_VERSION: 2.1.0-SNAPSHOT SPARK_VERSION_SHORT: 2.1.0 SCALA_BINARY_VERSION: "2.11" SCALA_VERSION: "2.11.7" -MESOS_VERSION: 0.22.0 +MESOS_VERSION: 1.0.0 SPARK_ISSUE_TRACKER_URL: https://issues.apache.org/jira/browse/SPARK SPARK_GITHUB_URL: https://github.com/apache/spark diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md index ce888b5445..d037e7be0a 100644 --- a/docs/running-on-mesos.md +++ b/docs/running-on-mesos.md @@ -432,6 +432,16 @@ See the [configuration page](configuration.html) for information on Spark config + + spark.mesos.containerizer + docker + + This only affects docker containers, and must be one of "docker" + or "mesos". Mesos supports two types of + containerizers for docker: the "docker" containerizer, and the preferred + "mesos" containerizer. Read more here: http://mesos.apache.org/documentation/latest/container-image/ + + spark.mesos.driver.webui.url (none) -- cgit v1.2.3