diff options
| author | Ivan Wick <ivanwick+github@gmail.com> | 2014-04-10 17:49:30 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-04-10 17:49:30 -0700 |
| commit | 5cd11d51c19321981a6234a7765c7a5be6913433 (patch) | |
| tree | 9df31b48f4561a6f1480f05ebabdbdf14ff2d891 /python/pyspark/shell.py | |
| parent | 2c557837b4a12c644cc37bd00d02be04f3807637 (diff) | |
| download | spark-5cd11d51c19321981a6234a7765c7a5be6913433.tar.gz spark-5cd11d51c19321981a6234a7765c7a5be6913433.tar.bz2 spark-5cd11d51c19321981a6234a7765c7a5be6913433.zip | |
Set spark.executor.uri from environment variable (needed by Mesos)
The Mesos backend uses this property when setting up a slave process. It is similarly set in the Scala repl (org.apache.spark.repl.SparkILoop), but I couldn't find any analogous for pyspark.
Author: Ivan Wick <ivanwick+github@gmail.com>
This patch had conflicts when merged, resolved by
Committer: Matei Zaharia <matei@databricks.com>
Closes #311 from ivanwick/master and squashes the following commits:
da0c3e4 [Ivan Wick] Set spark.executor.uri from environment variable (needed by Mesos)
Diffstat (limited to 'python/pyspark/shell.py')
| -rw-r--r-- | python/pyspark/shell.py | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py index 35e48276e3..61613dbed8 100644 --- a/python/pyspark/shell.py +++ b/python/pyspark/shell.py @@ -29,6 +29,9 @@ from pyspark.storagelevel import StorageLevel # this is the equivalent of ADD_JARS add_files = os.environ.get("ADD_FILES").split(',') if os.environ.get("ADD_FILES") != None else None +if os.environ.get("SPARK_EXECUTOR_URI"): + SparkContext.setSystemProperty("spark.executor.uri", os.environ["SPARK_EXECUTOR_URI"]) + sc = SparkContext(os.environ.get("MASTER", "local[*]"), "PySparkShell", pyFiles=add_files) print """Welcome to |