diff options
| author | KevinGrealish <KevinGre@microsoft.com> | 2016-07-27 13:50:31 -0700 |
|---|---|---|
| committer | Marcelo Vanzin <vanzin@cloudera.com> | 2016-07-27 13:50:31 -0700 |
| commit | b14d7b5cf4f173a1e45a4b1ae2a5e4e7ac5e9bb1 (patch) | |
| tree | c5bf2444ca8bee5fab106b9a10cd26bcd456d2fd /sql/core/src/test/scala | |
| parent | bc4851adeb386edc5bef47027a12ca44eda82b09 (diff) | |
| download | spark-b14d7b5cf4f173a1e45a4b1ae2a5e4e7ac5e9bb1.tar.gz spark-b14d7b5cf4f173a1e45a4b1ae2a5e4e7ac5e9bb1.tar.bz2 spark-b14d7b5cf4f173a1e45a4b1ae2a5e4e7ac5e9bb1.zip | |
[SPARK-16110][YARN][PYSPARK] Fix allowing python version to be specified per submit for cluster mode.
## What changes were proposed in this pull request?
This fix allows submit of pyspark jobs to specify python 2 or 3.
Change ordering in setup for application master environment so env vars PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON can be overridden by spark.yarn.appMasterEnv.* conf settings. This applies to YARN in cluster mode. This allows them to be set per submission without needing the unset the env vars (which is not always possible - e.g. batch submit with LIVY only exposes the arguments to spark-submit)
## How was this patch tested?
Manual and existing unit tests.
Author: KevinGrealish <KevinGre@microsoft.com>
Closes #13824 from KevinGrealish/SPARK-16110.
Diffstat (limited to 'sql/core/src/test/scala')
0 files changed, 0 insertions, 0 deletions