diff options
author | Daniel Darabos <darabos.daniel@gmail.com> | 2014-07-08 10:43:46 -0700 |
---|---|---|
committer | Aaron Davidson <aaron@databricks.com> | 2014-07-08 10:43:46 -0700 |
commit | c8a2313cdf825e0191680a423d17619b5504ff89 (patch) | |
tree | 0551408c058007c1b316f13dc85a48c729c2a8e8 /core | |
parent | cc3e0a14daf756ff5c2d4e7916438e175046e5bb (diff) | |
download | spark-c8a2313cdf825e0191680a423d17619b5504ff89.tar.gz spark-c8a2313cdf825e0191680a423d17619b5504ff89.tar.bz2 spark-c8a2313cdf825e0191680a423d17619b5504ff89.zip |
[SPARK-2403] Catch all errors during serialization in DAGScheduler
https://issues.apache.org/jira/browse/SPARK-2403
Spark hangs for us whenever we forget to register a class with Kryo. This should be a simple fix for that. But let me know if you have a better suggestion.
I did not write a new test for this. It would be pretty complicated and I'm not sure it's worthwhile for such a simple change. Let me know if you disagree.
Author: Daniel Darabos <darabos.daniel@gmail.com>
Closes #1329 from darabos/spark-2403 and squashes the following commits:
3aceaad [Daniel Darabos] Print full stack trace for miscellaneous exceptions during serialization.
52c22ba [Daniel Darabos] Only catch NonFatal exceptions.
361e962 [Daniel Darabos] Catch all errors during serialization in DAGScheduler.
Diffstat (limited to 'core')
-rw-r--r-- | core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala index 81c136d970..f72bfde572 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala @@ -26,6 +26,7 @@ import scala.concurrent.Await import scala.concurrent.duration._ import scala.language.postfixOps import scala.reflect.ClassTag +import scala.util.control.NonFatal import akka.actor._ import akka.actor.OneForOneStrategy @@ -768,6 +769,10 @@ class DAGScheduler( abortStage(stage, "Task not serializable: " + e.toString) runningStages -= stage return + case NonFatal(e) => // Other exceptions, such as IllegalArgumentException from Kryo. + abortStage(stage, s"Task serialization failed: $e\n${e.getStackTraceString}") + runningStages -= stage + return } logInfo("Submitting " + tasks.size + " missing tasks from " + stage + " (" + stage.rdd + ")") |