From db7113b1d37e86253d8584b88ed66672f3620254 Mon Sep 17 00:00:00 2001 From: Steve Loughran Date: Sun, 24 Apr 2016 10:32:22 +0100 Subject: [SPARK-13267][WEB UI] document the ?param arguments of the REST API; lift the… MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add to the REST API details on the ? args and examples from the test suite. I've used the existing table, adding all the fields to the second table. see [in the pr](https://github.com/steveloughran/spark/blob/history/SPARK-13267-doc-params/docs/monitoring.md). There's a slightly more sophisticated option: make the table 3 columns wide, and for all existing entries, have the initial `td` span 2 columns. The new entries would then have an empty 1st column, param in 2nd and text in 3rd, with any examples after a `br` entry. Author: Steve Loughran Closes #11152 from steveloughran/history/SPARK-13267-doc-params. --- docs/monitoring.md | 67 +++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 51 insertions(+), 16 deletions(-) diff --git a/docs/monitoring.md b/docs/monitoring.md index 9dcb0708d7..9912cde743 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -229,27 +229,46 @@ both running applications, and in the history server. The endpoints are mounted for the history server, they would typically be accessible at `http://:18080/api/v1`, and for a running application, at `http://localhost:4040/api/v1`. +In the API, an application is referenced by its application ID, `[app-id]`. +When running on YARN, each application may have multiple attempts; each identified by their `[attempt-id]`. +In the API listed below, `[app-id]` will actually be `[base-app-id]/[attempt-id]`, +where `[base-app-id]` is the YARN application ID. + - + - + - + - + - + @@ -257,36 +276,52 @@ for a running application, at `http://localhost:4040/api/v1`. - + - + - + - + - + - - + + - - + +
EndpointMeaning
/applicationsA list of all applicationsA list of all applications. +
+ ?status=[completed|running] list only applications in the chosen state. +
+ ?minDate=[date] earliest date/time to list. +
Examples: +
?minDate=2015-02-10 +
?minDate=2015-02-03T16:42:40.000GMT +
?maxDate=[date] latest date/time to list; uses same format as minDate.
/applications/[app-id]/jobsA list of all jobs for a given application + A list of all jobs for a given application. +
?status=[complete|succeeded|failed] list only jobs in the specific state. +
/applications/[app-id]/jobs/[job-id]Details for the given jobDetails for the given job.
/applications/[app-id]/stagesA list of all stages for a given applicationA list of all stages for a given application.
/applications/[app-id]/stages/[stage-id]A list of all attempts for the given stage + A list of all attempts for the given stage. +
?status=[active|complete|pending|failed] list only stages in the state. +
/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]
/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummarySummary metrics of all tasks in the given stage attempt + Summary metrics of all tasks in the given stage attempt. +
?quantiles summarize the metrics with the given quantiles. +
Example: ?quantiles=0.01,0.5,0.99 +
/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskListA list of all tasks for the given stage attempt + A list of all tasks for the given stage attempt. +
?offset=[offset]&length=[len] list tasks in the given range. +
?sortBy=[runtime|-runtime] sort the tasks. +
Example: ?offset=10&length=50&sortBy=runtime +
/applications/[app-id]/executorsA list of all executors for the given applicationA list of all executors for the given application.
/applications/[app-id]/storage/rddA list of stored RDDs for the given applicationA list of stored RDDs for the given application.
/applications/[app-id]/storage/rdd/[rdd-id]Details for the storage status of a given RDDDetails for the storage status of a given RDD.
/applications/[app-id]/logsDownload the event logs for all attempts of the given application as a zip file/applications/[base-app-id]/logsDownload the event logs for all attempts of the given application as files within + a zip file. +
/applications/[app-id]/[attempt-id]/logsDownload the event logs for the specified attempt of the given application as a zip file/applications/[base-app-id]/[attempt-id]/logsDownload the event logs for a specific application attempt as a zip file.
-When running on Yarn, each application has multiple attempts, so `[app-id]` is actually -`[app-id]/[attempt-id]` in all cases. +The number of jobs and stages which can retrieved is constrained by the same retention +mechanism of the standalone Spark UI; `"spark.ui.retainedJobs"` defines the threshold +value triggering garbage collection on jobs, and `spark.ui.retainedStages` that for stages. +Note that the garbage collection takes place on playback: it is possible to retrieve +more entries by increasing these values and restarting the history server. + +### API Versioning Policy These endpoints have been strongly versioned to make it easier to develop applications on top. In particular, Spark guarantees: -- cgit v1.2.3