diff options
author | Liang-Chi Hsieh <viirya@gmail.com> | 2017-02-17 11:28:16 -0800 |
---|---|---|
committer | Marcelo Vanzin <vanzin@cloudera.com> | 2017-02-17 11:28:16 -0800 |
commit | 4cc06f4eb1fbf1ba1fc6165783e22f93dc3b14ac (patch) | |
tree | 9822d0db176a0f02f18595fe0510ad3f26db31b8 /core/src/main/scala/org | |
parent | 3d0c3af0a76757c20e429c38efa4f14a15c9097a (diff) | |
download | spark-4cc06f4eb1fbf1ba1fc6165783e22f93dc3b14ac.tar.gz spark-4cc06f4eb1fbf1ba1fc6165783e22f93dc3b14ac.tar.bz2 spark-4cc06f4eb1fbf1ba1fc6165783e22f93dc3b14ac.zip |
[SPARK-18986][CORE] ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator
## What changes were proposed in this pull request?
`ExternalAppendOnlyMap.forceSpill` now uses an assert to check if an iterator is not null in the map. However, the assertion is only true after the map is asked for iterator. Before it, if another memory consumer asks more memory than currently available, `ExternalAppendOnlyMap.forceSpill` is also be called too. In this case, we will see failure like this:
[info] java.lang.AssertionError: assertion failed
[info] at scala.Predef$.assert(Predef.scala:156)
[info] at org.apache.spark.util.collection.ExternalAppendOnlyMap.forceSpill(ExternalAppendOnlyMap.scala:196)
[info] at org.apache.spark.util.collection.Spillable.spill(Spillable.scala:111)
[info] at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$13.apply$mcV$sp(ExternalAppendOnlyMapSuite.scala:294)
This fixing is motivated by http://apache-spark-developers-list.1001551.n3.nabble.com/java-lang-AssertionError-assertion-failed-tc20277.html.
## How was this patch tested?
Jenkins tests.
Please review http://spark.apache.org/contributing.html before opening a pull request.
Author: Liang-Chi Hsieh <viirya@gmail.com>
Closes #16387 from viirya/fix-externalappendonlymap.
Diffstat (limited to 'core/src/main/scala/org')
-rw-r--r-- | core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala | 17 |
1 files changed, 12 insertions, 5 deletions
diff --git a/core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala b/core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala index 948cc3b099..8aafda5e45 100644 --- a/core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala +++ b/core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala @@ -192,12 +192,19 @@ class ExternalAppendOnlyMap[K, V, C]( * It will be called by TaskMemoryManager when there is not enough memory for the task. */ override protected[this] def forceSpill(): Boolean = { - assert(readingIterator != null) - val isSpilled = readingIterator.spill() - if (isSpilled) { - currentMap = null + if (readingIterator != null) { + val isSpilled = readingIterator.spill() + if (isSpilled) { + currentMap = null + } + isSpilled + } else if (currentMap.size > 0) { + spill(currentMap) + currentMap = new SizeTrackingAppendOnlyMap[K, C] + true + } else { + false } - isSpilled } /** |