diff options
| author | Jeff Zhang <zjffdu@apache.org> | 2017-02-24 15:04:42 -0800 |
|---|---|---|
| committer | Holden Karau <holden@us.ibm.com> | 2017-02-24 15:04:42 -0800 |
| commit | 330c3e33bd10f035f49cf3d13357eb2d6d90dabc (patch) | |
| tree | 835b639982c7856a202af9100ab0e633069625fd /python/pyspark/context.py | |
| parent | 5f74148bb45912b9f867174de46e246215c93ee1 (diff) | |
| download | spark-330c3e33bd10f035f49cf3d13357eb2d6d90dabc.tar.gz spark-330c3e33bd10f035f49cf3d13357eb2d6d90dabc.tar.bz2 spark-330c3e33bd10f035f49cf3d13357eb2d6d90dabc.zip | |
[SPARK-13330][PYSPARK] PYTHONHASHSEED is not propgated to python worker
## What changes were proposed in this pull request?
self.environment will be propagated to executor. Should set PYTHONHASHSEED as long as the python version is greater than 3.3
## How was this patch tested?
Manually tested it.
Author: Jeff Zhang <zjffdu@apache.org>
Closes #11211 from zjffdu/SPARK-13330.
Diffstat (limited to 'python/pyspark/context.py')
| -rw-r--r-- | python/pyspark/context.py | 6 |
1 files changed, 2 insertions, 4 deletions
diff --git a/python/pyspark/context.py b/python/pyspark/context.py index ac4b2b035f..2961cda553 100644 --- a/python/pyspark/context.py +++ b/python/pyspark/context.py @@ -173,10 +173,8 @@ class SparkContext(object): if k.startswith("spark.executorEnv."): varName = k[len("spark.executorEnv."):] self.environment[varName] = v - if sys.version >= '3.3' and 'PYTHONHASHSEED' not in os.environ: - # disable randomness of hash of string in worker, if this is not - # launched by spark-submit - self.environment["PYTHONHASHSEED"] = "0" + + self.environment["PYTHONHASHSEED"] = os.environ.get("PYTHONHASHSEED", "0") # Create the Java SparkContext through Py4J self._jsc = jsc or self._initialize_context(self._conf._jconf) |