diff options
| author | CodingCat <zhunansjtu@gmail.com> | 2014-03-16 22:14:59 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-03-16 22:14:59 -0700 |
| commit | dc9654638f1d781ee1e54348fa41436b27793365 (patch) | |
| tree | 6b0278c8ca06dd78add6f035aad90d1608354549 /python | |
| parent | f5486e9f75d62919583da5ecf9a9ad00222b2227 (diff) | |
| download | spark-dc9654638f1d781ee1e54348fa41436b27793365.tar.gz spark-dc9654638f1d781ee1e54348fa41436b27793365.tar.bz2 spark-dc9654638f1d781ee1e54348fa41436b27793365.zip | |
SPARK-1240: handle the case of empty RDD when takeSample
https://spark-project.atlassian.net/browse/SPARK-1240
It seems that the current implementation does not handle the empty RDD case when run takeSample
In this patch, before calling sample() inside takeSample API, I add a checker for this case and returns an empty Array when it's a empty RDD; also in sample(), I add a checker for the invalid fraction value
In the test case, I also add several lines for this case
Author: CodingCat <zhunansjtu@gmail.com>
Closes #135 from CodingCat/SPARK-1240 and squashes the following commits:
fef57d4 [CodingCat] fix the same problem in PySpark
36db06b [CodingCat] create new test cases for takeSample from an empty red
810948d [CodingCat] further fix
a40e8fb [CodingCat] replace if with require
ad483fd [CodingCat] handle the case with empty RDD when take sample
Diffstat (limited to 'python')
| -rw-r--r-- | python/pyspark/rdd.py | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/python/pyspark/rdd.py b/python/pyspark/rdd.py index 6d549b40e5..f3b432ff24 100644 --- a/python/pyspark/rdd.py +++ b/python/pyspark/rdd.py @@ -268,6 +268,7 @@ class RDD(object): >>> sc.parallelize(range(0, 100)).sample(False, 0.1, 2).collect() #doctest: +SKIP [2, 3, 20, 21, 24, 41, 42, 66, 67, 89, 90, 98] """ + assert fraction >= 0.0, "Invalid fraction value: %s" % fraction return self.mapPartitionsWithIndex(RDDSampler(withReplacement, fraction, seed).func, True) # this is ported from scala/spark/RDD.scala @@ -288,6 +289,9 @@ class RDD(object): if (num < 0): raise ValueError + if (initialCount == 0): + return list() + if initialCount > sys.maxint - 1: maxSelected = sys.maxint - 1 else: |