diff options
| author | Iulian Dragos <jaguarul@gmail.com> | 2015-04-28 12:08:18 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-04-28 12:08:18 -0700 |
| commit | 8aab94d8984e9d12194dbda47b2e7d9dbc036889 (patch) | |
| tree | 6ae73a4d711bb4a44db690390ab7626ad2c40f61 /conf | |
| parent | 52ccf1d3739694826915cdf01642bab02958eb78 (diff) | |
| download | spark-8aab94d8984e9d12194dbda47b2e7d9dbc036889.tar.gz spark-8aab94d8984e9d12194dbda47b2e7d9dbc036889.tar.bz2 spark-8aab94d8984e9d12194dbda47b2e7d9dbc036889.zip | |
[SPARK-4286] Add an external shuffle service that can be run as a daemon.
This allows Mesos deployments to use the shuffle service (and implicitly dynamic allocation). It does so by adding a new "main" class and two corresponding scripts in `sbin`:
- `sbin/start-shuffle-service.sh`
- `sbin/stop-shuffle-service.sh`
Specific options can be passed in `SPARK_SHUFFLE_OPTS`.
This is picking up work from #3861 /cc tnachen
Author: Iulian Dragos <jaguarul@gmail.com>
Closes #4990 from dragos/feature/external-shuffle-service and squashes the following commits:
6c2b148 [Iulian Dragos] Import order and wrong name fixup.
07804ad [Iulian Dragos] Moved ExternalShuffleService to the `deploy` package + other minor tweaks.
4dc1f91 [Iulian Dragos] Reviewer’s comments:
8145429 [Iulian Dragos] Add an external shuffle service that can be run as a daemon.
Diffstat (limited to 'conf')
| -rwxr-xr-x | conf/spark-env.sh.template | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/conf/spark-env.sh.template b/conf/spark-env.sh.template index 67f81d3336..43c4288912 100755 --- a/conf/spark-env.sh.template +++ b/conf/spark-env.sh.template @@ -3,7 +3,7 @@ # This file is sourced when running various Spark programs. # Copy it as spark-env.sh and edit that to configure Spark for your site. -# Options read when launching programs locally with +# Options read when launching programs locally with # ./bin/run-example or ./bin/spark-submit # - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files # - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node @@ -39,6 +39,7 @@ # - SPARK_WORKER_DIR, to set the working directory of worker processes # - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y") # - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y") +# - SPARK_SHUFFLE_OPTS, to set config properties only for the external shuffle service (e.g. "-Dx=y") # - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y") # - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers |