diff options
| author | Jeff Zhang <zjffdu@apache.org> | 2017-02-24 15:04:42 -0800 |
|---|---|---|
| committer | Holden Karau <holden@us.ibm.com> | 2017-02-24 15:04:42 -0800 |
| commit | 330c3e33bd10f035f49cf3d13357eb2d6d90dabc (patch) | |
| tree | 835b639982c7856a202af9100ab0e633069625fd /python/pyspark/rdd.py | |
| parent | 5f74148bb45912b9f867174de46e246215c93ee1 (diff) | |
| download | spark-330c3e33bd10f035f49cf3d13357eb2d6d90dabc.tar.gz spark-330c3e33bd10f035f49cf3d13357eb2d6d90dabc.tar.bz2 spark-330c3e33bd10f035f49cf3d13357eb2d6d90dabc.zip | |
[SPARK-13330][PYSPARK] PYTHONHASHSEED is not propgated to python worker
## What changes were proposed in this pull request?
self.environment will be propagated to executor. Should set PYTHONHASHSEED as long as the python version is greater than 3.3
## How was this patch tested?
Manually tested it.
Author: Jeff Zhang <zjffdu@apache.org>
Closes #11211 from zjffdu/SPARK-13330.
Diffstat (limited to 'python/pyspark/rdd.py')
| -rw-r--r-- | python/pyspark/rdd.py | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/python/pyspark/rdd.py b/python/pyspark/rdd.py index b384b2b507..a5e6e2b054 100644 --- a/python/pyspark/rdd.py +++ b/python/pyspark/rdd.py @@ -68,7 +68,8 @@ def portable_hash(x): >>> portable_hash((None, 1)) & 0xffffffff 219750521 """ - if sys.version >= '3.3' and 'PYTHONHASHSEED' not in os.environ: + + if sys.version_info >= (3, 2, 3) and 'PYTHONHASHSEED' not in os.environ: raise Exception("Randomness of hash of string should be disabled via PYTHONHASHSEED") if x is None: |