diff options
author | Andrew Or <andrewor14@gmail.com> | 2014-10-30 15:29:07 -0700 |
---|---|---|
committer | Andrew Or <andrew@databricks.com> | 2014-10-30 15:29:07 -0700 |
commit | 24c5129257ce6e3b734f168e860b714c2730b55f (patch) | |
tree | 8c2e4e4a4e44a0d82b11f78a2b9d32c790d594e6 /yarn | |
parent | 9142c9b80bfe12e0be8a2b795bf52e403b2c5f30 (diff) | |
download | spark-24c5129257ce6e3b734f168e860b714c2730b55f.tar.gz spark-24c5129257ce6e3b734f168e860b714c2730b55f.tar.bz2 spark-24c5129257ce6e3b734f168e860b714c2730b55f.zip |
[SPARK-3319] [SPARK-3338] Resolve Spark submit config paths
The bulk of this PR is comprised of tests. All changes in functionality are made in `SparkSubmit.scala` (~20 lines).
**SPARK-3319.** There is currently a divergence in behavior when the user passes in additional jars through `--jars` and through setting `spark.jars` in the default properties file. The former will happily resolve the paths (e.g. convert `my.jar` to `file:/absolute/path/to/my.jar`), while the latter does not. We should resolve paths consistently in both cases. This also applies to the following pairs of command line arguments and Spark configs:
- `--jars` ~ `spark.jars`
- `--files` ~ `spark.files` / `spark.yarn.dist.files`
- `--archives` ~ `spark.yarn.dist.archives`
- `--py-files` ~ `spark.submit.pyFiles`
**SPARK-3338.** This PR also fixes the following bug: if the user sets `spark.submit.pyFiles` in his/her properties file, it does not actually get picked up even if `--py-files` is not set. This is simply because the config is overridden by an empty string.
Author: Andrew Or <andrewor14@gmail.com>
Author: Andrew Or <andrew@databricks.com>
Closes #2232 from andrewor14/resolve-config-paths and squashes the following commits:
fff2869 [Andrew Or] Add spark.yarn.jar
da3a1c1 [Andrew Or] Merge branch 'master' of github.com:apache/spark into resolve-config-paths
f0fae64 [Andrew Or] Merge branch 'master' of github.com:apache/spark into resolve-config-paths
05e03d6 [Andrew Or] Add tests for resolving both command line and config paths
460117e [Andrew Or] Resolve config paths properly
fe039d3 [Andrew Or] Beef up tests to test fixed-pointed-ness of Utils.resolveURI(s)
Diffstat (limited to 'yarn')
0 files changed, 0 insertions, 0 deletions