diff options
| author | Patrick Wendell <pwendell@gmail.com> | 2014-04-28 17:29:22 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-04-28 17:29:22 -0700 |
| commit | 949e393101e19cd00591a9930c4b364278e22609 (patch) | |
| tree | dcfa2c2337d25df917486f7f75771718a371055f /bin/spark-class | |
| parent | cae054aaf41ca0ee585231896db67169b61af689 (diff) | |
| download | spark-949e393101e19cd00591a9930c4b364278e22609.tar.gz spark-949e393101e19cd00591a9930c4b364278e22609.tar.bz2 spark-949e393101e19cd00591a9930c4b364278e22609.zip | |
SPARK-1654 and SPARK-1653: Fixes in spark-submit.
Deals with two issues:
1. Spark shell didn't correctly pass quoted arguments to spark-submit.
```./bin/spark-shell --driver-java-options "-Dfoo=f -Dbar=b"```
2. Spark submit used deprecated environment variables (SPARK_CLASSPATH)
which triggered warnings. Now we use new, more narrowly scoped,
variables.
Author: Patrick Wendell <pwendell@gmail.com>
Closes #576 from pwendell/spark-submit and squashes the following commits:
67004c9 [Patrick Wendell] SPARK-1654 and SPARK-1653: Fixes in spark-submit.
Diffstat (limited to 'bin/spark-class')
| -rwxr-xr-x | bin/spark-class | 9 |
1 files changed, 5 insertions, 4 deletions
diff --git a/bin/spark-class b/bin/spark-class index 6871e180c9..e8160c8af6 100755 --- a/bin/spark-class +++ b/bin/spark-class @@ -73,11 +73,13 @@ case "$1" in OUR_JAVA_MEM=${SPARK_EXECUTOR_MEMORY:-$DEFAULT_MEM} ;; - # All drivers use SPARK_JAVA_OPTS + SPARK_DRIVER_MEMORY. The repl also uses SPARK_REPL_OPTS. - 'org.apache.spark.repl.Main') - OUR_JAVA_OPTS="$SPARK_JAVA_OPTS $SPARK_REPL_OPTS" + # Spark submit uses SPARK_SUBMIT_OPTS and SPARK_JAVA_OPTS + 'org.apache.spark.deploy.SparkSubmit') + OUR_JAVA_OPTS="$SPARK_JAVA_OPTS $SPARK_SUBMIT_OPTS \ + -Djava.library.path=$SPARK_SUBMIT_LIBRARY_PATH" OUR_JAVA_MEM=${SPARK_DRIVER_MEMORY:-$DEFAULT_MEM} ;; + *) OUR_JAVA_OPTS="$SPARK_JAVA_OPTS" OUR_JAVA_MEM=${SPARK_DRIVER_MEMORY:-$DEFAULT_MEM} @@ -98,7 +100,6 @@ fi # Set JAVA_OPTS to be able to load native libraries and to set heap size JAVA_OPTS="$OUR_JAVA_OPTS" -JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$_SPARK_LIBRARY_PATH" JAVA_OPTS="$JAVA_OPTS -Xms$OUR_JAVA_MEM -Xmx$OUR_JAVA_MEM" # Load extra JAVA_OPTS from conf/java-opts, if it exists if [ -e "$FWDIR/conf/java-opts" ] ; then |