diff options
author | Xiangrui Meng <meng@databricks.com> | 2014-11-03 12:24:24 -0800 |
---|---|---|
committer | Xiangrui Meng <meng@databricks.com> | 2014-11-03 12:24:24 -0800 |
commit | 3cca1962207745814b9d83e791713c91b659c36c (patch) | |
tree | 997330407c25dabe70d5792421ce6e831976300b /python/pyspark/rddsampler.py | |
parent | 2aca97c7cfdefea8b6f9dbb88951e9acdfd606d9 (diff) | |
download | spark-3cca1962207745814b9d83e791713c91b659c36c.tar.gz spark-3cca1962207745814b9d83e791713c91b659c36c.tar.bz2 spark-3cca1962207745814b9d83e791713c91b659c36c.zip |
[SPARK-4148][PySpark] fix seed distribution and add some tests for rdd.sample
The current way of seed distribution makes the random sequences from partition i and i+1 offset by 1.
~~~
In [14]: import random
In [15]: r1 = random.Random(10)
In [16]: r1.randint(0, 1)
Out[16]: 1
In [17]: r1.random()
Out[17]: 0.4288890546751146
In [18]: r1.random()
Out[18]: 0.5780913011344704
In [19]: r2 = random.Random(10)
In [20]: r2.randint(0, 1)
Out[20]: 1
In [21]: r2.randint(0, 1)
Out[21]: 0
In [22]: r2.random()
Out[22]: 0.5780913011344704
~~~
Note: The new tests are not for this bug fix.
Author: Xiangrui Meng <meng@databricks.com>
Closes #3010 from mengxr/SPARK-4148 and squashes the following commits:
869ae4b [Xiangrui Meng] move tests tests.py
c1bacd9 [Xiangrui Meng] fix seed distribution and add some tests for rdd.sample
Diffstat (limited to 'python/pyspark/rddsampler.py')
-rw-r--r-- | python/pyspark/rddsampler.py | 11 |
1 files changed, 5 insertions, 6 deletions
diff --git a/python/pyspark/rddsampler.py b/python/pyspark/rddsampler.py index 528a181e89..f5c3cfd259 100644 --- a/python/pyspark/rddsampler.py +++ b/python/pyspark/rddsampler.py @@ -40,14 +40,13 @@ class RDDSamplerBase(object): def initRandomGenerator(self, split): if self._use_numpy: import numpy - self._random = numpy.random.RandomState(self._seed) + self._random = numpy.random.RandomState(self._seed ^ split) else: - self._random = random.Random(self._seed) + self._random = random.Random(self._seed ^ split) - for _ in range(0, split): - # discard the next few values in the sequence to have a - # different seed for the different splits - self._random.randint(0, 2 ** 32 - 1) + # mixing because the initial seeds are close to each other + for _ in xrange(10): + self._random.randint(0, 1) self._split = split self._rand_initialized = True |