diff options
| author | Jeff Zhang <zjffdu@apache.org> | 2017-02-24 15:04:42 -0800 |
|---|---|---|
| committer | Holden Karau <holden@us.ibm.com> | 2017-02-24 15:04:42 -0800 |
| commit | 330c3e33bd10f035f49cf3d13357eb2d6d90dabc (patch) | |
| tree | 835b639982c7856a202af9100ab0e633069625fd /core | |
| parent | 5f74148bb45912b9f867174de46e246215c93ee1 (diff) | |
| download | spark-330c3e33bd10f035f49cf3d13357eb2d6d90dabc.tar.gz spark-330c3e33bd10f035f49cf3d13357eb2d6d90dabc.tar.bz2 spark-330c3e33bd10f035f49cf3d13357eb2d6d90dabc.zip | |
[SPARK-13330][PYSPARK] PYTHONHASHSEED is not propgated to python worker
## What changes were proposed in this pull request?
self.environment will be propagated to executor. Should set PYTHONHASHSEED as long as the python version is greater than 3.3
## How was this patch tested?
Manually tested it.
Author: Jeff Zhang <zjffdu@apache.org>
Closes #11211 from zjffdu/SPARK-13330.
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala b/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala index 0b1cec2df8..a8f732b11f 100644 --- a/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala +++ b/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala @@ -85,6 +85,7 @@ object PythonRunner { // pass conf spark.pyspark.python to python process, the only way to pass info to // python process is through environment variable. sparkConf.get(PYSPARK_PYTHON).foreach(env.put("PYSPARK_PYTHON", _)) + sys.env.get("PYTHONHASHSEED").foreach(env.put("PYTHONHASHSEED", _)) builder.redirectErrorStream(true) // Ugly but needed for stdout and stderr to synchronize try { val process = builder.start() |