diff options
| author | WangTaoTheTonic <barneystinson@aliyun.com> | 2015-01-09 17:10:02 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-01-09 17:10:02 -0800 |
| commit | 8782eb992f461502238c41ece3a3002efa67a792 (patch) | |
| tree | fb04d788145745dd58fb948c4248b47806f15a70 /bin | |
| parent | 4e1f12d997426560226648d62ee17c90352613e7 (diff) | |
| download | spark-8782eb992f461502238c41ece3a3002efa67a792.tar.gz spark-8782eb992f461502238c41ece3a3002efa67a792.tar.bz2 spark-8782eb992f461502238c41ece3a3002efa67a792.zip | |
[SPARK-4990][Deploy]to find default properties file, search SPARK_CONF_DIR first
https://issues.apache.org/jira/browse/SPARK-4990
Author: WangTaoTheTonic <barneystinson@aliyun.com>
Author: WangTao <barneystinson@aliyun.com>
Closes #3823 from WangTaoTheTonic/SPARK-4990 and squashes the following commits:
133c43e [WangTao] Update spark-submit2.cmd
b1ab402 [WangTao] Update spark-submit
4cc7f34 [WangTaoTheTonic] rebase
55300bc [WangTaoTheTonic] use export to make it global
d8d3cb7 [WangTaoTheTonic] remove blank line
07b9ebf [WangTaoTheTonic] check SPARK_CONF_DIR instead of checking properties file
c5a85eb [WangTaoTheTonic] to find default properties file, search SPARK_CONF_DIR first
Diffstat (limited to 'bin')
| -rwxr-xr-x | bin/spark-submit | 5 | ||||
| -rw-r--r-- | bin/spark-submit2.cmd | 6 |
2 files changed, 9 insertions, 2 deletions
diff --git a/bin/spark-submit b/bin/spark-submit index aefd38a0a2..3e5cbdbb24 100755 --- a/bin/spark-submit +++ b/bin/spark-submit @@ -44,7 +44,10 @@ while (($#)); do shift done -DEFAULT_PROPERTIES_FILE="$SPARK_HOME/conf/spark-defaults.conf" +if [ -z "$SPARK_CONF_DIR" ]; then + export SPARK_CONF_DIR="$SPARK_HOME/conf" +fi +DEFAULT_PROPERTIES_FILE="$SPARK_CONF_DIR/spark-defaults.conf" if [ "$MASTER" == "yarn-cluster" ]; then SPARK_SUBMIT_DEPLOY_MODE=cluster fi diff --git a/bin/spark-submit2.cmd b/bin/spark-submit2.cmd index daf0284db9..12244a9cb0 100644 --- a/bin/spark-submit2.cmd +++ b/bin/spark-submit2.cmd @@ -24,7 +24,11 @@ set ORIG_ARGS=%* rem Reset the values of all variables used set SPARK_SUBMIT_DEPLOY_MODE=client -set SPARK_SUBMIT_PROPERTIES_FILE=%SPARK_HOME%\conf\spark-defaults.conf + +if not defined %SPARK_CONF_DIR% ( + set SPARK_CONF_DIR=%SPARK_HOME%\conf +) +set SPARK_SUBMIT_PROPERTIES_FILE=%SPARK_CONF_DIR%\spark-defaults.conf set SPARK_SUBMIT_DRIVER_MEMORY= set SPARK_SUBMIT_LIBRARY_PATH= set SPARK_SUBMIT_CLASSPATH= |