diff options
| author | moussa taifi <moutai10@gmail.com> | 2015-02-26 14:19:43 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-02-26 14:20:36 -0800 |
| commit | 297c3ef826933b4a71d5a54950cabdb1d7d54613 (patch) | |
| tree | 8d7070fd4a5754a6f828f7ba99b85d4bcc0ea9b6 /docs/monitoring.md | |
| parent | fe7967483d1b0871c128863f51826c43fd71a12e (diff) | |
| download | spark-297c3ef826933b4a71d5a54950cabdb1d7d54613.tar.gz spark-297c3ef826933b4a71d5a54950cabdb1d7d54613.tar.bz2 spark-297c3ef826933b4a71d5a54950cabdb1d7d54613.zip | |
Add a note for context termination for History server on Yarn
The history server on Yarn only shows completed jobs. This adds a note concerning the needed explicit context termination at the end of a spark job which is a best practice anyway.
Related to SPARK-2972 and SPARK-3458
Author: moussa taifi <moutai10@gmail.com>
Closes #4721 from moutai/add-history-server-note-for-closing-the-spark-context and squashes the following commits:
9f5b6c3 [moussa taifi] Fix upper case typo for YARN
3ad3db4 [moussa taifi] Add context termination for History server on Yarn
(cherry picked from commit c871e2dae0182e914135560d14304242e1f97f7e)
Signed-off-by: Andrew Or <andrew@databricks.com>
Diffstat (limited to 'docs/monitoring.md')
| -rw-r--r-- | docs/monitoring.md | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/docs/monitoring.md b/docs/monitoring.md index 7a5cadc171..657bf67be2 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -150,6 +150,8 @@ follows: Note that in all of these UIs, the tables are sortable by clicking their headers, making it easy to identify slow tasks, data skew, etc. +Note that the history server only displays completed Spark jobs. One way to signal the completion of a Spark job is to stop the Spark Context explicitly (`sc.stop()`), or in Python using the `with SparkContext() as sc:` to handle the Spark Context setup and tear down, and still show the job history on the UI. + # Metrics Spark has a configurable metrics system based on the |