diff options
author | Yash Datta <Yash.Datta@guavus.com> | 2014-12-29 13:49:45 -0800 |
---|---|---|
committer | Reynold Xin <rxin@databricks.com> | 2014-12-29 13:49:45 -0800 |
commit | 9bc0df6804f241aff24520d9c6ec54d9b11f5785 (patch) | |
tree | df233fb21efa57191b33083834348e59124ab189 /mllib | |
parent | 02b55de3dce9a1fef806be13e5cefa0f39ea2fcc (diff) | |
download | spark-9bc0df6804f241aff24520d9c6ec54d9b11f5785.tar.gz spark-9bc0df6804f241aff24520d9c6ec54d9b11f5785.tar.bz2 spark-9bc0df6804f241aff24520d9c6ec54d9b11f5785.zip |
SPARK-4968: takeOrdered to skip reduce step in case mappers return no partitions
takeOrdered should skip reduce step in case mapped RDDs have no partitions. This prevents the mentioned exception :
4. run query
SELECT * FROM testTable WHERE market = 'market2' ORDER BY End_Time DESC LIMIT 100;
Error trace
java.lang.UnsupportedOperationException: empty collection
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:863)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:863)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.reduce(RDD.scala:863)
at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1136)
Author: Yash Datta <Yash.Datta@guavus.com>
Closes #3830 from saucam/fix_takeorder and squashes the following commits:
5974d10 [Yash Datta] SPARK-4968: takeOrdered to skip reduce step in case mappers return no partitions
Diffstat (limited to 'mllib')
0 files changed, 0 insertions, 0 deletions