diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2015-04-08 10:14:52 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-04-08 10:15:10 -0700 |
| commit | e967ecacad8075ef521fbc1a501e074c861d0fe7 (patch) | |
| tree | 8d58d23be629b1836ac176d3e17147280ee886aa | |
| parent | cdef7d080aa3f473f5ea06ba816c01b41a0239eb (diff) | |
| download | spark-e967ecacad8075ef521fbc1a501e074c861d0fe7.tar.gz spark-e967ecacad8075ef521fbc1a501e074c861d0fe7.tar.bz2 spark-e967ecacad8075ef521fbc1a501e074c861d0fe7.zip | |
[SPARK-6506] [pyspark] Do not try to retrieve SPARK_HOME when not needed...
....
In particular, this makes pyspark in yarn-cluster mode fail unless
SPARK_HOME is set, when it's not really needed.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #5405 from vanzin/SPARK-6506 and squashes the following commits:
e184507 [Marcelo Vanzin] [SPARK-6506] [pyspark] Do not try to retrieve SPARK_HOME when not needed.
(cherry picked from commit f7e21dd1ec4541be54eb01d8b15cfcc6714feed0)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
| -rw-r--r-- | python/pyspark/java_gateway.py | 3 |
1 files changed, 1 insertions, 2 deletions
diff --git a/python/pyspark/java_gateway.py b/python/pyspark/java_gateway.py index 936857e75c..19ee2e3add 100644 --- a/python/pyspark/java_gateway.py +++ b/python/pyspark/java_gateway.py @@ -29,11 +29,10 @@ from pyspark.serializers import read_int def launch_gateway(): - SPARK_HOME = os.environ["SPARK_HOME"] - if "PYSPARK_GATEWAY_PORT" in os.environ: gateway_port = int(os.environ["PYSPARK_GATEWAY_PORT"]) else: + SPARK_HOME = os.environ["SPARK_HOME"] # Launch the Py4j gateway using Spark's run command so that we pick up the # proper classpath and settings from spark-env.sh on_windows = platform.system() == "Windows" |