diff options
| author | Burak Yavuz <brkyvz@gmail.com> | 2015-02-17 17:15:43 -0800 |
|---|---|---|
| committer | Patrick Wendell <patrick@databricks.com> | 2015-02-17 17:23:22 -0800 |
| commit | ae6cfb3acdbc2721d25793698a4a440f0519dbec (patch) | |
| tree | 4dba3eaff24a4d042ac6e9e0a3e1b8c5c6108f14 /docs/submitting-applications.md | |
| parent | c3d2b90bde2e11823909605d518167548df66bd8 (diff) | |
| download | spark-ae6cfb3acdbc2721d25793698a4a440f0519dbec.tar.gz spark-ae6cfb3acdbc2721d25793698a4a440f0519dbec.tar.bz2 spark-ae6cfb3acdbc2721d25793698a4a440f0519dbec.zip | |
[SPARK-5811] Added documentation for maven coordinates and added Spark Packages support
Documentation for maven coordinates + Spark Package support. Added pyspark tests for `--packages`
Author: Burak Yavuz <brkyvz@gmail.com>
Author: Davies Liu <davies@databricks.com>
Closes #4662 from brkyvz/SPARK-5811 and squashes the following commits:
56ccccd [Burak Yavuz] fixed broken test
64cb8ee [Burak Yavuz] passed pep8 on local
c07b81e [Burak Yavuz] fixed pep8
a8bd6b7 [Burak Yavuz] submit PR
4ef4046 [Burak Yavuz] ready for PR
8fb02e5 [Burak Yavuz] merged master
25c9b9f [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into python-jar
560d13b [Burak Yavuz] before PR
17d3f76 [Davies Liu] support .jar as python package
a3eb717 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into SPARK-5811
c60156d [Burak Yavuz] [SPARK-5811] Added documentation for maven coordinates
Diffstat (limited to 'docs/submitting-applications.md')
| -rw-r--r-- | docs/submitting-applications.md | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/docs/submitting-applications.md b/docs/submitting-applications.md index 14a87f8436..57b074778f 100644 --- a/docs/submitting-applications.md +++ b/docs/submitting-applications.md @@ -174,6 +174,11 @@ This can use up a significant amount of space over time and will need to be clea is handled automatically, and with Spark standalone, automatic cleanup can be configured with the `spark.worker.cleanup.appDataTtl` property. +Users may also include any other dependencies by supplying a comma-delimited list of maven coordinates +with `--packages`. All transitive dependencies will be handled when using this command. Additional +repositories (or resolvers in SBT) can be added in a comma-delimited fashion with the flag `--repositories`. +These commands can be used with `pyspark`, `spark-shell`, and `spark-submit` to include Spark Packages. + For Python, the equivalent `--py-files` option can be used to distribute `.egg`, `.zip` and `.py` libraries to executors. |