diff options
| author | Guillaume Poulin <poulin.guillaume@gmail.com> | 2016-04-05 02:54:38 +0100 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-04-05 02:54:38 +0100 |
| commit | 7201f033ce520259b6d07ea5ead92272cac92363 (patch) | |
| tree | 294fdda0323bef5a42a7f3c3b5be5b1a66bd7eb2 /core | |
| parent | a172e11cba6f917baf5bd6c4f83dc6689932de9a (diff) | |
| download | spark-7201f033ce520259b6d07ea5ead92272cac92363.tar.gz spark-7201f033ce520259b6d07ea5ead92272cac92363.tar.bz2 spark-7201f033ce520259b6d07ea5ead92272cac92363.zip | |
[SPARK-12425][STREAMING] DStream union optimisation
Use PartitionerAwareUnionRDD when possbile for optimizing shuffling and
preserving the partitioner.
Author: Guillaume Poulin <poulin.guillaume@gmail.com>
Closes #10382 from gpoulin/dstream_union_optimisation.
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/rdd/RDD.scala | 6 |
1 files changed, 1 insertions, 5 deletions
diff --git a/core/src/main/scala/org/apache/spark/rdd/RDD.scala b/core/src/main/scala/org/apache/spark/rdd/RDD.scala index 4a0a2199ef..032939b49a 100644 --- a/core/src/main/scala/org/apache/spark/rdd/RDD.scala +++ b/core/src/main/scala/org/apache/spark/rdd/RDD.scala @@ -568,11 +568,7 @@ abstract class RDD[T: ClassTag]( * times (use `.distinct()` to eliminate them). */ def union(other: RDD[T]): RDD[T] = withScope { - if (partitioner.isDefined && other.partitioner == partitioner) { - new PartitionerAwareUnionRDD(sc, Array(this, other)) - } else { - new UnionRDD(sc, Array(this, other)) - } + sc.union(this, other) } /** |