diff options
| author | Holden Karau <holden@pigscanfly.ca> | 2015-07-01 23:05:45 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-07-01 23:05:57 -0700 |
| commit | e33c0f0a497194d93b3c034502a9a49dc22c0cdf (patch) | |
| tree | 4b8357155faeb74e73e8847d541b9213b9c50a12 | |
| parent | 5b468cf0c21071d212b0cba7a0cede7eeb5d273b (diff) | |
| download | spark-e33c0f0a497194d93b3c034502a9a49dc22c0cdf.tar.gz spark-e33c0f0a497194d93b3c034502a9a49dc22c0cdf.tar.bz2 spark-e33c0f0a497194d93b3c034502a9a49dc22c0cdf.zip | |
[SPARK-8769] [TRIVIAL] [DOCS] toLocalIterator should mention it results in many jobs
Author: Holden Karau <holden@pigscanfly.ca>
Closes #7171 from holdenk/SPARK-8769-toLocalIterator-documentation-improvement and squashes the following commits:
97ddd99 [Holden Karau] Add note
(cherry picked from commit 15d41cc501f5fa7ac82c4a6741e416bb557f610a)
Signed-off-by: Andrew Or <andrew@databricks.com>
| -rw-r--r-- | core/src/main/scala/org/apache/spark/rdd/RDD.scala | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/rdd/RDD.scala b/core/src/main/scala/org/apache/spark/rdd/RDD.scala index 10610f4b6f..cac6e3b477 100644 --- a/core/src/main/scala/org/apache/spark/rdd/RDD.scala +++ b/core/src/main/scala/org/apache/spark/rdd/RDD.scala @@ -890,6 +890,10 @@ abstract class RDD[T: ClassTag]( * Return an iterator that contains all of the elements in this RDD. * * The iterator will consume as much memory as the largest partition in this RDD. + * + * Note: this results in multiple Spark jobs, and if the input RDD is the result + * of a wide transformation (e.g. join with different partitioners), to avoid + * recomputing the input RDD should be cached first. */ def toLocalIterator: Iterator[T] = withScope { def collectPartition(p: Int): Array[T] = { |