aboutsummaryrefslogtreecommitdiff
path: root/docs/cluster-overview.md
diff options
context:
space:
mode:
authorPatrick Wendell <pwendell@gmail.com>2014-04-26 19:24:29 -0700
committerPatrick Wendell <pwendell@gmail.com>2014-04-26 19:24:29 -0700
commitaa9a7f5db7bffcdbcd7fd53694c606d3a8cdd21f (patch)
tree2dd20bd11dd1673677c2b52a9aa3b5774b4f3c45 /docs/cluster-overview.md
parent762af4e9c2837c8ca125838b2ca01e328ee6669f (diff)
downloadspark-aa9a7f5db7bffcdbcd7fd53694c606d3a8cdd21f.tar.gz
spark-aa9a7f5db7bffcdbcd7fd53694c606d3a8cdd21f.tar.bz2
spark-aa9a7f5db7bffcdbcd7fd53694c606d3a8cdd21f.zip
SPARK-1606: Infer user application arguments instead of requiring --arg.
This modifies spark-submit to do something more like the Hadoop `jar` command. Now we have the following syntax: ./bin/spark-submit [options] user.jar [user options] Author: Patrick Wendell <pwendell@gmail.com> Closes #563 from pwendell/spark-submit and squashes the following commits: 32241fc [Patrick Wendell] Review feedback 3adfb69 [Patrick Wendell] Small fix bc48139 [Patrick Wendell] SPARK-1606: Infer user application arguments instead of requiring --arg.
Diffstat (limited to 'docs/cluster-overview.md')
-rw-r--r--docs/cluster-overview.md22
1 files changed, 13 insertions, 9 deletions
diff --git a/docs/cluster-overview.md b/docs/cluster-overview.md
index dcc0630426..b011679fed 100644
--- a/docs/cluster-overview.md
+++ b/docs/cluster-overview.md
@@ -73,30 +73,34 @@ the bin directory. This script takes care of setting up the classpath with Spark
dependencies, and can support different cluster managers and deploy modes that Spark supports.
It's usage is
- ./bin/spark-submit <app jar> --class path.to.your.Class [other options..]
+ ./bin/spark-submit --class path.to.your.Class [options] <app jar> [app options]
-To enumerate all options available to `spark-submit` run it with the `--help` flag.
-Here are a few examples of common options:
+When calling `spark-submit`, `[app options]` will be passed along to your application's
+main class. To enumerate all options available to `spark-submit` run it with
+the `--help` flag. Here are a few examples of common options:
{% highlight bash %}
# Run application locally
-./bin/spark-submit my-app.jar \
+./bin/spark-submit \
--class my.main.ClassName
- --master local[8]
+ --master local[8] \
+ my-app.jar
# Run on a Spark cluster
-./bin/spark-submit my-app.jar \
+./bin/spark-submit \
--class my.main.ClassName
--master spark://mycluster:7077 \
--executor-memory 20G \
- --total-executor-cores 100
+ --total-executor-cores 100 \
+ my-app.jar
# Run on a YARN cluster
-HADOOP_CONF_DIR=XX /bin/spark-submit my-app.jar \
+HADOOP_CONF_DIR=XX /bin/spark-submit \
--class my.main.ClassName
--master yarn-cluster \ # can also be `yarn-client` for client mode
--executor-memory 20G \
- --num-executors 50
+ --num-executors 50 \
+ my-app.jar
{% endhighlight %}
### Loading Configurations from a File