diff options
| author | moussa taifi <moutai10@gmail.com> | 2015-02-26 14:19:43 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-02-26 14:20:30 -0800 |
| commit | c871e2dae0182e914135560d14304242e1f97f7e (patch) | |
| tree | a4df9c08035ff0fbacdcdaf03938b5606a693998 /docs/monitoring.md | |
| parent | 3fb53c0298761ba227890525ae79ce4ec6300deb (diff) | |
| download | spark-c871e2dae0182e914135560d14304242e1f97f7e.tar.gz spark-c871e2dae0182e914135560d14304242e1f97f7e.tar.bz2 spark-c871e2dae0182e914135560d14304242e1f97f7e.zip | |
Add a note for context termination for History server on Yarn
The history server on Yarn only shows completed jobs. This adds a note concerning the needed explicit context termination at the end of a spark job which is a best practice anyway.
Related to SPARK-2972 and SPARK-3458
Author: moussa taifi <moutai10@gmail.com>
Closes #4721 from moutai/add-history-server-note-for-closing-the-spark-context and squashes the following commits:
9f5b6c3 [moussa taifi] Fix upper case typo for YARN
3ad3db4 [moussa taifi] Add context termination for History server on Yarn
Diffstat (limited to 'docs/monitoring.md')
| -rw-r--r-- | docs/monitoring.md | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/docs/monitoring.md b/docs/monitoring.md index 37ede476c1..6816671ffb 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -173,6 +173,8 @@ follows: Note that in all of these UIs, the tables are sortable by clicking their headers, making it easy to identify slow tasks, data skew, etc. +Note that the history server only displays completed Spark jobs. One way to signal the completion of a Spark job is to stop the Spark Context explicitly (`sc.stop()`), or in Python using the `with SparkContext() as sc:` to handle the Spark Context setup and tear down, and still show the job history on the UI. + # Metrics Spark has a configurable metrics system based on the |