diff options
author | Andrew Or <andrewor14@gmail.com> | 2014-05-22 20:32:27 -0700 |
---|---|---|
committer | Tathagata Das <tathagata.das1565@gmail.com> | 2014-05-22 20:32:43 -0700 |
commit | c3b406518788e084f2cfb723294eec543ef4c4fb (patch) | |
tree | 5b2cc9cc608af60d3f8a9449d2291ec9327d7de4 /repl | |
parent | 23cc40e39acc598816ac46b36463e13587a0dd60 (diff) | |
download | spark-c3b406518788e084f2cfb723294eec543ef4c4fb.tar.gz spark-c3b406518788e084f2cfb723294eec543ef4c4fb.tar.bz2 spark-c3b406518788e084f2cfb723294eec543ef4c4fb.zip |
[SPARK-1896] Respect spark.master (and --master) before MASTER in spark-shell
The hierarchy for configuring the Spark master in the shell is as follows:
```
MASTER > --master > spark.master (spark-defaults.conf)
```
This is inconsistent with the way we run normal applications, which is:
```
--master > spark.master (spark-defaults.conf) > MASTER
```
I was trying to run a shell locally on a standalone cluster launched through the ec2 scripts, which automatically set `MASTER` in spark-env.sh. It was surprising to me that `--master` didn't take effect, considering that this is the way we tell users to set their masters [here](http://people.apache.org/~pwendell/spark-1.0.0-rc7-docs/scala-programming-guide.html#initializing-spark).
Author: Andrew Or <andrewor14@gmail.com>
Closes #846 from andrewor14/shell-master and squashes the following commits:
2cb81c9 [Andrew Or] Respect spark.master before MASTER in REPL
(cherry picked from commit cce77457e00aa5f1f4db3d50454cf257efb156ed)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Diffstat (limited to 'repl')
-rw-r--r-- | repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala | 5 |
1 files changed, 2 insertions, 3 deletions
diff --git a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala b/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala index 55684e94bd..5f34362ccd 100644 --- a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala +++ b/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala @@ -962,11 +962,10 @@ class SparkILoop(in0: Option[BufferedReader], protected val out: JPrintWriter, private def getMaster(): String = { val master = this.master match { case Some(m) => m - case None => { + case None => val envMaster = sys.env.get("MASTER") val propMaster = sys.props.get("spark.master") - envMaster.orElse(propMaster).getOrElse("local[*]") - } + propMaster.orElse(envMaster).getOrElse("local[*]") } master } |