diff options
| author | Lianhui Wang <lianhuiwang09@gmail.com> | 2015-07-16 19:31:14 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-07-16 19:31:45 -0700 |
| commit | 49351c7f597c67950cc65e5014a89fad31b9a6f7 (patch) | |
| tree | 887471f1d7c0f9efd3e7e9f864c9e0c7ff6fda1c /python/pyspark/context.py | |
| parent | 57e9b13bf99ab82896114d47c62fbee648c9be7d (diff) | |
| download | spark-49351c7f597c67950cc65e5014a89fad31b9a6f7.tar.gz spark-49351c7f597c67950cc65e5014a89fad31b9a6f7.tar.bz2 spark-49351c7f597c67950cc65e5014a89fad31b9a6f7.zip | |
[SPARK-8646] PySpark does not run on YARN if master not provided in command line
andrewor14 davies vanzin can you take a look at this? thanks
Author: Lianhui Wang <lianhuiwang09@gmail.com>
Closes #7438 from lianhuiwang/SPARK-8646 and squashes the following commits:
cb3f12d [Lianhui Wang] add whitespace
6d874a6 [Lianhui Wang] support pyspark for yarn-client
Diffstat (limited to 'python/pyspark/context.py')
| -rw-r--r-- | python/pyspark/context.py | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/python/pyspark/context.py b/python/pyspark/context.py index d7466729b8..43bde5ae41 100644 --- a/python/pyspark/context.py +++ b/python/pyspark/context.py @@ -152,6 +152,11 @@ class SparkContext(object): self.master = self._conf.get("spark.master") self.appName = self._conf.get("spark.app.name") self.sparkHome = self._conf.get("spark.home", None) + + # Let YARN know it's a pyspark app, so it distributes needed libraries. + if self.master == "yarn-client": + self._conf.set("spark.yarn.isPython", "true") + for (k, v) in self._conf.getAll(): if k.startswith("spark.executorEnv."): varName = k[len("spark.executorEnv."):] |