diff options
| author | Miao Wang <miaowang@Miaos-MacBook-Pro.local> | 2016-10-27 01:17:32 +0200 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-10-27 01:17:32 +0200 |
| commit | a76846cfb1c2d6c8f4d647426030b59de20d9433 (patch) | |
| tree | 05b5b45003adaff5dde2862c5ee3fcca503145ba /core/src/test/scala | |
| parent | 29cea8f332aa3750f8ff7c3b9e705d107278da4b (diff) | |
| download | spark-a76846cfb1c2d6c8f4d647426030b59de20d9433.tar.gz spark-a76846cfb1c2d6c8f4d647426030b59de20d9433.tar.bz2 spark-a76846cfb1c2d6c8f4d647426030b59de20d9433.zip | |
[SPARK-18126][SPARK-CORE] getIteratorZipWithIndex accepts negative value as index
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
`Utils.getIteratorZipWithIndex` was added to deal with number of records > 2147483647 in one partition.
method `getIteratorZipWithIndex` accepts `startIndex` < 0, which leads to negative index.
This PR just adds a defensive check on `startIndex` to make sure it is >= 0.
## How was this patch tested?
Add a new unit test.
Author: Miao Wang <miaowang@Miaos-MacBook-Pro.local>
Closes #15639 from wangmiao1981/zip.
Diffstat (limited to 'core/src/test/scala')
| -rw-r--r-- | core/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala index aeb2969fd5..15ef32f21d 100644 --- a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala +++ b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala @@ -401,6 +401,9 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties with Logging { assert(iterator.toArray === Array( (0, -1L + Int.MaxValue), (1, 0L + Int.MaxValue), (2, 1L + Int.MaxValue) )) + intercept[IllegalArgumentException] { + Utils.getIteratorZipWithIndex(Iterator(0, 1, 2), -1L) + } } test("doesDirectoryContainFilesNewerThan") { |