diff options
| author | jerryshao <saisai.shao@intel.com> | 2014-09-29 11:25:32 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-09-29 11:25:43 -0700 |
| commit | df5a62f510031cf2be20f0f7c6ff33d82233359a (patch) | |
| tree | 3d0b7ab31f871458baa6ad06b2c438f00d923827 /core/src/main | |
| parent | 7d88471e894d8dd825f89a7320523a10826e741b (diff) | |
| download | spark-df5a62f510031cf2be20f0f7c6ff33d82233359a.tar.gz spark-df5a62f510031cf2be20f0f7c6ff33d82233359a.tar.bz2 spark-df5a62f510031cf2be20f0f7c6ff33d82233359a.zip | |
[SPARK-3032][Shuffle] Fix key comparison integer overflow introduced sorting exception
Previous key comparison in `ExternalSorter` will get wrong sorting result or exception when key comparison overflows, details can be seen in [SPARK-3032](https://issues.apache.org/jira/browse/SPARK-3032). Here fix this and add a unit test to prove it.
Author: jerryshao <saisai.shao@intel.com>
Closes #2514 from jerryshao/SPARK-3032 and squashes the following commits:
6f3c302 [jerryshao] Improve the unit test according to comments
01911e6 [jerryshao] Change the test to show the contract violate exception
83acb38 [jerryshao] Minor changes according to comments
fa2a08f [jerryshao] Fix key comparison integer overflow introduced sorting exception
(cherry picked from commit dab1b0ae29a6d3017bdca23464f22a51d51eaae1)
Signed-off-by: Matei Zaharia <matei@databricks.com>
Diffstat (limited to 'core/src/main')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala b/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala index 5d8a648d95..b58c7dd984 100644 --- a/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala +++ b/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala @@ -152,7 +152,7 @@ private[spark] class ExternalSorter[K, V, C]( override def compare(a: K, b: K): Int = { val h1 = if (a == null) 0 else a.hashCode() val h2 = if (b == null) 0 else b.hashCode() - h1 - h2 + if (h1 < h2) -1 else if (h1 == h2) 0 else 1 } }) |