diff options
| author | Jeff Zhang <zjffdu@apache.org> | 2016-03-16 10:21:01 -0700 |
|---|---|---|
| committer | Marcelo Vanzin <vanzin@cloudera.com> | 2016-03-16 10:21:01 -0700 |
| commit | eacd9d8eda68260bbda7b0cd07410321dffaf428 (patch) | |
| tree | fe972a34bad5274113e5e4853fedd8759e9caa64 /yarn | |
| parent | 5f6bdf97c5d2d78db0ca39703084d8ae377346b8 (diff) | |
| download | spark-eacd9d8eda68260bbda7b0cd07410321dffaf428.tar.gz spark-eacd9d8eda68260bbda7b0cd07410321dffaf428.tar.bz2 spark-eacd9d8eda68260bbda7b0cd07410321dffaf428.zip | |
[SPARK-13360][PYSPARK][YARN] PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON…
… is not picked up in yarn-cluster mode
Author: Jeff Zhang <zjffdu@apache.org>
Closes #11238 from zjffdu/SPARK-13360.
Diffstat (limited to 'yarn')
| -rw-r--r-- | yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala index 40cf9b68ed..f864ef13e0 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala @@ -745,6 +745,9 @@ private[spark] class Client( } env("SPARK_JAVA_OPTS") = value } + // propagate PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON to driver in cluster mode + sys.env.get("PYSPARK_DRIVER_PYTHON").foreach(env("PYSPARK_DRIVER_PYTHON") = _) + sys.env.get("PYSPARK_PYTHON").foreach(env("PYSPARK_PYTHON") = _) } sys.env.get(ENV_DIST_CLASSPATH).foreach { dcp => |