diff options
author | Timothy Chen <tnachen@gmail.com> | 2015-08-03 01:55:58 -0700 |
---|---|---|
committer | Andrew Or <andrew@databricks.com> | 2015-08-03 01:55:58 -0700 |
commit | 95dccc63350c45045f038bab9f8a5080b4e1f8cc (patch) | |
tree | c6ac9b266234a3d349ce4ada3a80a17adc7744fc /sbin/start-mesos-shuffle-service.sh | |
parent | 1ebd41b141a95ec264bd2dd50f0fe24cd459035d (diff) | |
download | spark-95dccc63350c45045f038bab9f8a5080b4e1f8cc.tar.gz spark-95dccc63350c45045f038bab9f8a5080b4e1f8cc.tar.bz2 spark-95dccc63350c45045f038bab9f8a5080b4e1f8cc.zip |
[SPARK-8873] [MESOS] Clean up shuffle files if external shuffle service is used
This patch builds directly on #7820, which is largely written by tnachen. The only addition is one commit for cleaning up the code. There should be no functional differences between this and #7820.
Author: Timothy Chen <tnachen@gmail.com>
Author: Andrew Or <andrew@databricks.com>
Closes #7881 from andrewor14/tim-cleanup-mesos-shuffle and squashes the following commits:
8894f7d [Andrew Or] Clean up code
2a5fa10 [Andrew Or] Merge branch 'mesos_shuffle_clean' of github.com:tnachen/spark into tim-cleanup-mesos-shuffle
fadff89 [Timothy Chen] Address comments.
e4d0f1d [Timothy Chen] Clean up external shuffle data on driver exit with Mesos.
Diffstat (limited to 'sbin/start-mesos-shuffle-service.sh')
-rwxr-xr-x | sbin/start-mesos-shuffle-service.sh | 35 |
1 files changed, 35 insertions, 0 deletions
diff --git a/sbin/start-mesos-shuffle-service.sh b/sbin/start-mesos-shuffle-service.sh new file mode 100755 index 0000000000..64580762c5 --- /dev/null +++ b/sbin/start-mesos-shuffle-service.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Starts the Mesos external shuffle server on the machine this script is executed on. +# The Mesos external shuffle service detects when an application exits and automatically +# cleans up its shuffle files. +# +# Usage: start-mesos-shuffle-server.sh +# +# Use the SPARK_SHUFFLE_OPTS environment variable to set shuffle service configuration. +# + +sbin="`dirname "$0"`" +sbin="`cd "$sbin"; pwd`" + +. "$sbin/spark-config.sh" +. "$SPARK_PREFIX/bin/load-spark-env.sh" + +exec "$sbin"/spark-daemon.sh start org.apache.spark.deploy.mesos.MesosExternalShuffleService 1 |