aboutsummaryrefslogtreecommitdiff
path: root/yarn
diff options
context:
space:
mode:
authorAndrew Or <andrewor14@gmail.com>2014-10-30 15:29:07 -0700
committerAndrew Or <andrew@databricks.com>2014-10-30 15:29:07 -0700
commit24c5129257ce6e3b734f168e860b714c2730b55f (patch)
tree8c2e4e4a4e44a0d82b11f78a2b9d32c790d594e6 /yarn
parent9142c9b80bfe12e0be8a2b795bf52e403b2c5f30 (diff)
downloadspark-24c5129257ce6e3b734f168e860b714c2730b55f.tar.gz
spark-24c5129257ce6e3b734f168e860b714c2730b55f.tar.bz2
spark-24c5129257ce6e3b734f168e860b714c2730b55f.zip
[SPARK-3319] [SPARK-3338] Resolve Spark submit config paths
The bulk of this PR is comprised of tests. All changes in functionality are made in `SparkSubmit.scala` (~20 lines). **SPARK-3319.** There is currently a divergence in behavior when the user passes in additional jars through `--jars` and through setting `spark.jars` in the default properties file. The former will happily resolve the paths (e.g. convert `my.jar` to `file:/absolute/path/to/my.jar`), while the latter does not. We should resolve paths consistently in both cases. This also applies to the following pairs of command line arguments and Spark configs: - `--jars` ~ `spark.jars` - `--files` ~ `spark.files` / `spark.yarn.dist.files` - `--archives` ~ `spark.yarn.dist.archives` - `--py-files` ~ `spark.submit.pyFiles` **SPARK-3338.** This PR also fixes the following bug: if the user sets `spark.submit.pyFiles` in his/her properties file, it does not actually get picked up even if `--py-files` is not set. This is simply because the config is overridden by an empty string. Author: Andrew Or <andrewor14@gmail.com> Author: Andrew Or <andrew@databricks.com> Closes #2232 from andrewor14/resolve-config-paths and squashes the following commits: fff2869 [Andrew Or] Add spark.yarn.jar da3a1c1 [Andrew Or] Merge branch 'master' of github.com:apache/spark into resolve-config-paths f0fae64 [Andrew Or] Merge branch 'master' of github.com:apache/spark into resolve-config-paths 05e03d6 [Andrew Or] Add tests for resolving both command line and config paths 460117e [Andrew Or] Resolve config paths properly fe039d3 [Andrew Or] Beef up tests to test fixed-pointed-ness of Utils.resolveURI(s)
Diffstat (limited to 'yarn')
0 files changed, 0 insertions, 0 deletions