diff options
| author | Yash Datta <Yash.Datta@guavus.com> | 2014-12-29 13:49:45 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2014-12-29 13:49:45 -0800 |
| commit | 9bc0df6804f241aff24520d9c6ec54d9b11f5785 (patch) | |
| tree | df233fb21efa57191b33083834348e59124ab189 /examples | |
| parent | 02b55de3dce9a1fef806be13e5cefa0f39ea2fcc (diff) | |
| download | spark-9bc0df6804f241aff24520d9c6ec54d9b11f5785.tar.gz spark-9bc0df6804f241aff24520d9c6ec54d9b11f5785.tar.bz2 spark-9bc0df6804f241aff24520d9c6ec54d9b11f5785.zip | |
SPARK-4968: takeOrdered to skip reduce step in case mappers return no partitions
takeOrdered should skip reduce step in case mapped RDDs have no partitions. This prevents the mentioned exception :
4. run query
SELECT * FROM testTable WHERE market = 'market2' ORDER BY End_Time DESC LIMIT 100;
Error trace
java.lang.UnsupportedOperationException: empty collection
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:863)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:863)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.reduce(RDD.scala:863)
at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1136)
Author: Yash Datta <Yash.Datta@guavus.com>
Closes #3830 from saucam/fix_takeorder and squashes the following commits:
5974d10 [Yash Datta] SPARK-4968: takeOrdered to skip reduce step in case mappers return no partitions
Diffstat (limited to 'examples')
0 files changed, 0 insertions, 0 deletions