diff options
| author | peng.zhang <peng.zhang@xiaomi.com> | 2016-06-24 08:28:32 +0100 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-06-24 08:28:32 +0100 |
| commit | f4fd7432fb9cf7b197ccada1378c4f2a6d427522 (patch) | |
| tree | d63d5af02e5d87ecde98e891b278c467279d8ba6 /python | |
| parent | 2d2f607bfae97f2681df24f48bb8b1b483c6b309 (diff) | |
| download | spark-f4fd7432fb9cf7b197ccada1378c4f2a6d427522.tar.gz spark-f4fd7432fb9cf7b197ccada1378c4f2a6d427522.tar.bz2 spark-f4fd7432fb9cf7b197ccada1378c4f2a6d427522.zip | |
[SPARK-16125][YARN] Fix not test yarn cluster mode correctly in YarnClusterSuite
## What changes were proposed in this pull request?
Since SPARK-13220(Deprecate "yarn-client" and "yarn-cluster"), YarnClusterSuite doesn't test "yarn cluster" mode correctly.
This pull request fixes it.
## How was this patch tested?
Unit test
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Author: peng.zhang <peng.zhang@xiaomi.com>
Closes #13836 from renozhang/SPARK-16125-test-yarn-cluster-mode.
Diffstat (limited to 'python')
| -rw-r--r-- | python/pyspark/context.py | 4 |
1 files changed, 0 insertions, 4 deletions
diff --git a/python/pyspark/context.py b/python/pyspark/context.py index aec0215b40..7217a9907a 100644 --- a/python/pyspark/context.py +++ b/python/pyspark/context.py @@ -155,10 +155,6 @@ class SparkContext(object): self.appName = self._conf.get("spark.app.name") self.sparkHome = self._conf.get("spark.home", None) - # Let YARN know it's a pyspark app, so it distributes needed libraries. - if self.master == "yarn-client": - self._conf.set("spark.yarn.isPython", "true") - for (k, v) in self._conf.getAll(): if k.startswith("spark.executorEnv."): varName = k[len("spark.executorEnv."):] |