diff options
| author | Davies Liu <davies@databricks.com> | 2015-03-16 16:26:55 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-03-16 16:26:55 -0700 |
| commit | e3f315ac358dfe4f5b9705c3eac76e8b1e24f82a (patch) | |
| tree | bb196e4f6d31ed2b98abb03ca0c5da1cafcc8b99 /python/pyspark/java_gateway.py | |
| parent | f149b8b5e542af44650923d0156f037121b45a20 (diff) | |
| download | spark-e3f315ac358dfe4f5b9705c3eac76e8b1e24f82a.tar.gz spark-e3f315ac358dfe4f5b9705c3eac76e8b1e24f82a.tar.bz2 spark-e3f315ac358dfe4f5b9705c3eac76e8b1e24f82a.zip | |
[SPARK-6327] [PySpark] fix launch spark-submit from python
SparkSubmit should be launched without setting PYSPARK_SUBMIT_ARGS
cc JoshRosen , this mode is actually used by python unit test, so I will not add more test for it.
Author: Davies Liu <davies@databricks.com>
Closes #5019 from davies/fix_submit and squashes the following commits:
2c20b0c [Davies Liu] fix launch spark-submit from python
Diffstat (limited to 'python/pyspark/java_gateway.py')
| -rw-r--r-- | python/pyspark/java_gateway.py | 6 |
1 files changed, 2 insertions, 4 deletions
diff --git a/python/pyspark/java_gateway.py b/python/pyspark/java_gateway.py index 43d2cf5171..0a16cbd8bf 100644 --- a/python/pyspark/java_gateway.py +++ b/python/pyspark/java_gateway.py @@ -38,10 +38,8 @@ def launch_gateway(): # proper classpath and settings from spark-env.sh on_windows = platform.system() == "Windows" script = "./bin/spark-submit.cmd" if on_windows else "./bin/spark-submit" - submit_args = os.environ.get("PYSPARK_SUBMIT_ARGS") - submit_args = submit_args if submit_args is not None else "" - submit_args = shlex.split(submit_args) - command = [os.path.join(SPARK_HOME, script)] + submit_args + submit_args = os.environ.get("PYSPARK_SUBMIT_ARGS", "pyspark-shell") + command = [os.path.join(SPARK_HOME, script)] + shlex.split(submit_args) # Start a socket that will be used by PythonGatewayServer to communicate its port to us callback_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |