diff options
author | Patrick Wendell <pwendell@gmail.com> | 2014-04-26 19:24:29 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-04-26 19:24:29 -0700 |
commit | aa9a7f5db7bffcdbcd7fd53694c606d3a8cdd21f (patch) | |
tree | 2dd20bd11dd1673677c2b52a9aa3b5774b4f3c45 /docs/cluster-overview.md | |
parent | 762af4e9c2837c8ca125838b2ca01e328ee6669f (diff) | |
download | spark-aa9a7f5db7bffcdbcd7fd53694c606d3a8cdd21f.tar.gz spark-aa9a7f5db7bffcdbcd7fd53694c606d3a8cdd21f.tar.bz2 spark-aa9a7f5db7bffcdbcd7fd53694c606d3a8cdd21f.zip |
SPARK-1606: Infer user application arguments instead of requiring --arg.
This modifies spark-submit to do something more like the Hadoop `jar`
command. Now we have the following syntax:
./bin/spark-submit [options] user.jar [user options]
Author: Patrick Wendell <pwendell@gmail.com>
Closes #563 from pwendell/spark-submit and squashes the following commits:
32241fc [Patrick Wendell] Review feedback
3adfb69 [Patrick Wendell] Small fix
bc48139 [Patrick Wendell] SPARK-1606: Infer user application arguments instead of requiring --arg.
Diffstat (limited to 'docs/cluster-overview.md')
-rw-r--r-- | docs/cluster-overview.md | 22 |
1 files changed, 13 insertions, 9 deletions
diff --git a/docs/cluster-overview.md b/docs/cluster-overview.md index dcc0630426..b011679fed 100644 --- a/docs/cluster-overview.md +++ b/docs/cluster-overview.md @@ -73,30 +73,34 @@ the bin directory. This script takes care of setting up the classpath with Spark dependencies, and can support different cluster managers and deploy modes that Spark supports. It's usage is - ./bin/spark-submit <app jar> --class path.to.your.Class [other options..] + ./bin/spark-submit --class path.to.your.Class [options] <app jar> [app options] -To enumerate all options available to `spark-submit` run it with the `--help` flag. -Here are a few examples of common options: +When calling `spark-submit`, `[app options]` will be passed along to your application's +main class. To enumerate all options available to `spark-submit` run it with +the `--help` flag. Here are a few examples of common options: {% highlight bash %} # Run application locally -./bin/spark-submit my-app.jar \ +./bin/spark-submit \ --class my.main.ClassName - --master local[8] + --master local[8] \ + my-app.jar # Run on a Spark cluster -./bin/spark-submit my-app.jar \ +./bin/spark-submit \ --class my.main.ClassName --master spark://mycluster:7077 \ --executor-memory 20G \ - --total-executor-cores 100 + --total-executor-cores 100 \ + my-app.jar # Run on a YARN cluster -HADOOP_CONF_DIR=XX /bin/spark-submit my-app.jar \ +HADOOP_CONF_DIR=XX /bin/spark-submit \ --class my.main.ClassName --master yarn-cluster \ # can also be `yarn-client` for client mode --executor-memory 20G \ - --num-executors 50 + --num-executors 50 \ + my-app.jar {% endhighlight %} ### Loading Configurations from a File |