diff options
| author | pankaj arora <pankaj.arora@guavus.com> | 2015-04-14 12:06:46 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-04-14 12:07:08 -0700 |
| commit | dcf8a9f331c6193a62bbc9282bdc99663e23ca19 (patch) | |
| tree | 12b71a590c1972cc8a8a27f7ed253f7344004e20 | |
| parent | f63b44a5c201d9678738a906462be9a6d7e3e8f8 (diff) | |
| download | spark-dcf8a9f331c6193a62bbc9282bdc99663e23ca19.tar.gz spark-dcf8a9f331c6193a62bbc9282bdc99663e23ca19.tar.bz2 spark-dcf8a9f331c6193a62bbc9282bdc99663e23ca19.zip | |
[CORE] SPARK-6880: Fixed null check when all the dependent stages are cancelled due to previous stage failure
Fixed null check when all the dependent stages are cancelled due to previous stage failure. This happens when one of the executor node goes down and all the dependent stages are cancelled.
Author: pankaj arora <pankaj.arora@guavus.com>
Closes #5494 from pankajarora12/NEWBRANCH and squashes the following commits:
55ba5e3 [pankaj arora] [CORE] SPARK-6880: Fixed null check when all the dependent stages are cancelled due to previous stage failure
4575720 [pankaj arora] [CORE] SPARK-6880: Fixed null check when all the dependent stages are cancelled due to previous stage failure
| -rw-r--r-- | core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala | 7 |
1 files changed, 1 insertions, 6 deletions
diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala index 508fe7b330..4a32f8936f 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala @@ -818,12 +818,7 @@ class DAGScheduler( } } - val properties = if (jobIdToActiveJob.contains(jobId)) { - jobIdToActiveJob(stage.jobId).properties - } else { - // this stage will be assigned to "default" pool - null - } + val properties = jobIdToActiveJob.get(stage.jobId).map(_.properties).orNull runningStages += stage // SparkListenerStageSubmitted should be posted before testing whether tasks are |