diff options
author | Kay Ousterhout <kayousterhout@gmail.com> | 2014-07-25 15:14:13 -0700 |
---|---|---|
committer | Matei Zaharia <matei@databricks.com> | 2014-07-25 15:14:13 -0700 |
commit | 37ad3b724590dcf42bcdbfaf91b7a11914501945 (patch) | |
tree | ab1f5016c2b56aa48219878eb2aca64723b0aaba /external/kafka | |
parent | 47b6b38ca8d9c5de794183cc91cbf6559ef27390 (diff) | |
download | spark-37ad3b724590dcf42bcdbfaf91b7a11914501945.tar.gz spark-37ad3b724590dcf42bcdbfaf91b7a11914501945.tar.bz2 spark-37ad3b724590dcf42bcdbfaf91b7a11914501945.zip |
[SPARK-1726] [SPARK-2567] Eliminate zombie stages in UI.
Due to problems with when we update runningStages (in DAGScheduler.scala)
and how we decide to send a SparkListenerStageCompleted message to
SparkListeners, sometimes stages can be shown as "running" in the UI forever
(even after they have failed). This issue can manifest when stages are
resubmitted with 0 tasks, or when the DAGScheduler catches non-serializable
tasks. The problem also resulted in a (small) memory leak in the DAGScheduler,
where stages can stay in runningStages forever. This commit fixes
that problem and adds a unit test.
Thanks tsudukim for helping to look into this issue!
cc markhamstra rxin
Author: Kay Ousterhout <kayousterhout@gmail.com>
Closes #1566 from kayousterhout/dag_fix and squashes the following commits:
217d74b [Kay Ousterhout] [SPARK-1726] [SPARK-2567] Eliminate zombie stages in UI.
Diffstat (limited to 'external/kafka')
0 files changed, 0 insertions, 0 deletions