diff options
author | Milan Straka <fox@ucw.cz> | 2015-04-10 13:50:32 -0700 |
---|---|---|
committer | Josh Rosen <joshrosen@databricks.com> | 2015-04-10 13:50:32 -0700 |
commit | 0375134f42197f2e29aa865a513cda381f0a1445 (patch) | |
tree | ce4bfd1a4f9168b11bb183c6361503f9e9eafac6 /python/pyspark/worker.py | |
parent | b9baa4cd9f6e9fc58161f79744b6b7729894d920 (diff) | |
download | spark-0375134f42197f2e29aa865a513cda381f0a1445.tar.gz spark-0375134f42197f2e29aa865a513cda381f0a1445.tar.bz2 spark-0375134f42197f2e29aa865a513cda381f0a1445.zip |
[SPARK-5969][PySpark] Fix descending pyspark.rdd.sortByKey.
The samples should always be sorted in ascending order, because bisect.bisect_left is used on it. The reverse order of the result is already achieved in rangePartitioner by reversing the found index.
The current implementation also work, but always uses only two partitions -- the first one and the last one (because the bisect_left return returns either "beginning" or "end" for a descending sequence).
Author: Milan Straka <fox@ucw.cz>
This patch had conflicts when merged, resolved by
Committer: Josh Rosen <joshrosen@databricks.com>
Closes #4761 from foxik/fix-descending-sort and squashes the following commits:
95896b5 [Milan Straka] Add regression test for SPARK-5969.
5757490 [Milan Straka] Fix descending pyspark.rdd.sortByKey.
Diffstat (limited to 'python/pyspark/worker.py')
0 files changed, 0 insertions, 0 deletions