diff options
| author | jerryshao <saisai.shao@intel.com> | 2014-09-29 11:25:32 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-09-29 11:25:32 -0700 |
| commit | dab1b0ae29a6d3017bdca23464f22a51d51eaae1 (patch) | |
| tree | a8b35717e7a9e2c5e8526de8856aa4cc8d6c6f5e /core/src/main/scala | |
| parent | 587a0cd7ed964ebfca2c97924c4f1e363f1fd3cb (diff) | |
| download | spark-dab1b0ae29a6d3017bdca23464f22a51d51eaae1.tar.gz spark-dab1b0ae29a6d3017bdca23464f22a51d51eaae1.tar.bz2 spark-dab1b0ae29a6d3017bdca23464f22a51d51eaae1.zip | |
[SPARK-3032][Shuffle] Fix key comparison integer overflow introduced sorting exception
Previous key comparison in `ExternalSorter` will get wrong sorting result or exception when key comparison overflows, details can be seen in [SPARK-3032](https://issues.apache.org/jira/browse/SPARK-3032). Here fix this and add a unit test to prove it.
Author: jerryshao <saisai.shao@intel.com>
Closes #2514 from jerryshao/SPARK-3032 and squashes the following commits:
6f3c302 [jerryshao] Improve the unit test according to comments
01911e6 [jerryshao] Change the test to show the contract violate exception
83acb38 [jerryshao] Minor changes according to comments
fa2a08f [jerryshao] Fix key comparison integer overflow introduced sorting exception
Diffstat (limited to 'core/src/main/scala')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala b/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala index 0a152cb97a..644fa36818 100644 --- a/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala +++ b/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala @@ -144,7 +144,7 @@ private[spark] class ExternalSorter[K, V, C]( override def compare(a: K, b: K): Int = { val h1 = if (a == null) 0 else a.hashCode() val h2 = if (b == null) 0 else b.hashCode() - h1 - h2 + if (h1 < h2) -1 else if (h1 == h2) 0 else 1 } }) |