diff options
author | Davies Liu <davies@databricks.com> | 2015-03-16 16:26:55 -0700 |
---|---|---|
committer | Josh Rosen <joshrosen@databricks.com> | 2015-03-16 16:26:55 -0700 |
commit | e3f315ac358dfe4f5b9705c3eac76e8b1e24f82a (patch) | |
tree | bb196e4f6d31ed2b98abb03ca0c5da1cafcc8b99 | |
parent | f149b8b5e542af44650923d0156f037121b45a20 (diff) | |
download | spark-e3f315ac358dfe4f5b9705c3eac76e8b1e24f82a.tar.gz spark-e3f315ac358dfe4f5b9705c3eac76e8b1e24f82a.tar.bz2 spark-e3f315ac358dfe4f5b9705c3eac76e8b1e24f82a.zip |
[SPARK-6327] [PySpark] fix launch spark-submit from python
SparkSubmit should be launched without setting PYSPARK_SUBMIT_ARGS
cc JoshRosen , this mode is actually used by python unit test, so I will not add more test for it.
Author: Davies Liu <davies@databricks.com>
Closes #5019 from davies/fix_submit and squashes the following commits:
2c20b0c [Davies Liu] fix launch spark-submit from python
-rwxr-xr-x | bin/pyspark | 1 | ||||
-rw-r--r-- | python/pyspark/java_gateway.py | 6 |
2 files changed, 2 insertions, 5 deletions
diff --git a/bin/pyspark b/bin/pyspark index e7f6a1a072..776b28dc41 100755 --- a/bin/pyspark +++ b/bin/pyspark @@ -89,7 +89,6 @@ export PYTHONSTARTUP="$SPARK_HOME/python/pyspark/shell.py" if [[ -n "$SPARK_TESTING" ]]; then unset YARN_CONF_DIR unset HADOOP_CONF_DIR - export PYSPARK_SUBMIT_ARGS=pyspark-shell if [[ -n "$PYSPARK_DOC_TEST" ]]; then exec "$PYSPARK_DRIVER_PYTHON" -m doctest $1 else diff --git a/python/pyspark/java_gateway.py b/python/pyspark/java_gateway.py index 43d2cf5171..0a16cbd8bf 100644 --- a/python/pyspark/java_gateway.py +++ b/python/pyspark/java_gateway.py @@ -38,10 +38,8 @@ def launch_gateway(): # proper classpath and settings from spark-env.sh on_windows = platform.system() == "Windows" script = "./bin/spark-submit.cmd" if on_windows else "./bin/spark-submit" - submit_args = os.environ.get("PYSPARK_SUBMIT_ARGS") - submit_args = submit_args if submit_args is not None else "" - submit_args = shlex.split(submit_args) - command = [os.path.join(SPARK_HOME, script)] + submit_args + submit_args = os.environ.get("PYSPARK_SUBMIT_ARGS", "pyspark-shell") + command = [os.path.join(SPARK_HOME, script)] + shlex.split(submit_args) # Start a socket that will be used by PythonGatewayServer to communicate its port to us callback_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |