diff options
author | Davies Liu <davies@databricks.com> | 2014-11-18 16:37:35 -0800 |
---|---|---|
committer | Xiangrui Meng <meng@databricks.com> | 2014-11-18 16:37:43 -0800 |
commit | 70d9e3871f852ec9e8bfaa436bc02bc22fc62dfd (patch) | |
tree | 189f02d8ed0cbe6959ad74997335dbe95d35ffb9 /python/pyspark/broadcast.py | |
parent | bf76164f1090892544983f753d4b7b16903a6135 (diff) | |
download | spark-70d9e3871f852ec9e8bfaa436bc02bc22fc62dfd.tar.gz spark-70d9e3871f852ec9e8bfaa436bc02bc22fc62dfd.tar.bz2 spark-70d9e3871f852ec9e8bfaa436bc02bc22fc62dfd.zip |
[SPARK-4327] [PySpark] Python API for RDD.randomSplit()
```
pyspark.RDD.randomSplit(self, weights, seed=None)
Randomly splits this RDD with the provided weights.
:param weights: weights for splits, will be normalized if they don't sum to 1
:param seed: random seed
:return: split RDDs in an list
>>> rdd = sc.parallelize(range(10), 1)
>>> rdd1, rdd2, rdd3 = rdd.randomSplit([0.4, 0.6, 1.0], 11)
>>> rdd1.collect()
[3, 6]
>>> rdd2.collect()
[0, 5, 7]
>>> rdd3.collect()
[1, 2, 4, 8, 9]
```
Author: Davies Liu <davies@databricks.com>
Closes #3193 from davies/randomSplit and squashes the following commits:
78bf997 [Davies Liu] fix tests, do not use numpy in randomSplit, no performance gain
f5fdf63 [Davies Liu] fix bug with int in weights
4dfa2cd [Davies Liu] refactor
f866bcf [Davies Liu] remove unneeded change
c7a2007 [Davies Liu] switch to python implementation
95a48ac [Davies Liu] Merge branch 'master' of github.com:apache/spark into randomSplit
0d9b256 [Davies Liu] refactor
1715ee3 [Davies Liu] address comments
41fce54 [Davies Liu] randomSplit()
(cherry picked from commit 7f22fa81ebd5e501fcb0e1da5506d1d4fb9250cf)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
Diffstat (limited to 'python/pyspark/broadcast.py')
0 files changed, 0 insertions, 0 deletions