diff options
| author | Kay Ousterhout <kayousterhout@gmail.com> | 2014-07-25 15:14:13 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-07-25 15:14:13 -0700 |
| commit | 37ad3b724590dcf42bcdbfaf91b7a11914501945 (patch) | |
| tree | ab1f5016c2b56aa48219878eb2aca64723b0aaba /bin/beeline | |
| parent | 47b6b38ca8d9c5de794183cc91cbf6559ef27390 (diff) | |
| download | spark-37ad3b724590dcf42bcdbfaf91b7a11914501945.tar.gz spark-37ad3b724590dcf42bcdbfaf91b7a11914501945.tar.bz2 spark-37ad3b724590dcf42bcdbfaf91b7a11914501945.zip | |
[SPARK-1726] [SPARK-2567] Eliminate zombie stages in UI.
Due to problems with when we update runningStages (in DAGScheduler.scala)
and how we decide to send a SparkListenerStageCompleted message to
SparkListeners, sometimes stages can be shown as "running" in the UI forever
(even after they have failed). This issue can manifest when stages are
resubmitted with 0 tasks, or when the DAGScheduler catches non-serializable
tasks. The problem also resulted in a (small) memory leak in the DAGScheduler,
where stages can stay in runningStages forever. This commit fixes
that problem and adds a unit test.
Thanks tsudukim for helping to look into this issue!
cc markhamstra rxin
Author: Kay Ousterhout <kayousterhout@gmail.com>
Closes #1566 from kayousterhout/dag_fix and squashes the following commits:
217d74b [Kay Ousterhout] [SPARK-1726] [SPARK-2567] Eliminate zombie stages in UI.
Diffstat (limited to 'bin/beeline')
0 files changed, 0 insertions, 0 deletions