From 200f01c8fb15680b5630fbd122d44f9b1d096e02 Mon Sep 17 00:00:00 2001 From: Marcelo Vanzin Date: Tue, 7 Jun 2016 09:28:39 -0700 Subject: [SPARK-15760][DOCS] Add documentation for package-related configs. While there, also document spark.files and spark.jars. Text is the same as the spark-submit help text with some minor adjustments. Author: Marcelo Vanzin Closes #13502 from vanzin/SPARK-15760. --- docs/configuration.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index d6471a8cc7..32c3a92660 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -380,6 +380,53 @@ Apart from these, the following properties are also available, and may be useful from JVM to Python worker for every task. + + spark.files + + + Comma-separated list of files to be placed in the working directory of each executor. + + + + spark.submit.pyFiles + + + Comma-separated list of .zip, .egg, or .py files to place on the PYTHONPATH for Python apps. + + + + spark.jars + + + Comma-separated list of local jars to include on the driver and executor classpaths. + + + + spark.jars.packages + + + Comma-separated list of maven coordinates of jars to include on the driver and executor + classpaths. Will search the local maven repo, then maven central and any additional remote + repositories given by spark.jars.ivy. The format for the coordinates should be + groupId:artifactId:version. + + + + spark.jars.excludes + + + Comma-separated list of groupId:artifactId, to exclude while resolving the dependencies + provided in spark.jars.packages to avoid dependency conflicts. + + + + spark.jars.ivy + + + Comma-separated list of additional remote repositories to search for the coordinates given + with spark.jars.packages. + + #### Shuffle Behavior -- cgit v1.2.3