From 51b3d41e160a1326a04536241b427e65b39ed8df Mon Sep 17 00:00:00 2001 From: Reynold Xin Date: Tue, 5 May 2015 19:27:30 -0700 Subject: Revert "[SPARK-3454] separate json endpoints for data in the UI" This reverts commit d49735800db27239c11478aac4b0f2ec9df91a3f. The commit broke Spark on Windows. --- docs/monitoring.md | 74 ------------------------------------------------------ 1 file changed, 74 deletions(-) (limited to 'docs/monitoring.md') diff --git a/docs/monitoring.md b/docs/monitoring.md index 1e0fc15086..8a85928d6d 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -174,80 +174,6 @@ making it easy to identify slow tasks, data skew, etc. Note that the history server only displays completed Spark jobs. One way to signal the completion of a Spark job is to stop the Spark Context explicitly (`sc.stop()`), or in Python using the `with SparkContext() as sc:` to handle the Spark Context setup and tear down, and still show the job history on the UI. -## REST API - -In addition to viewing the metrics in the UI, they are also available as JSON. This gives developers -an easy way to create new visualizations and monitoring tools for Spark. The JSON is available for -both running applications, and in the history server. The endpoints are mounted at `/json/v1`. Eg., -for the history server, they would typically be accessible at `http://:18080/json/v1`, and -for a running application, at `http://localhost:4040/json/v1`. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
EndpointMeaning
/applicationsA list of all applications
/applications/[app-id]/jobsA list of all jobs for a given application
/applications/[app-id]/jobs/[job-id]Details for the given job
/applications/[app-id]/stagesA list of all stages for a given application
/applications/[app-id]/stages/[stage-id]A list of all attempts for the given stage
/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]Details for the given stage attempt
/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummarySummary metrics of all tasks in the given stage attempt
/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskListA list of all tasks for the given stage attempt
/applications/[app-id]/executorsA list of all executors for the given application
/applications/[app-id]/storage/rddA list of stored RDDs for the given application
/applications/[app-id]/storage/rdd/[rdd-id]Details for the storage status of a given RDD
- -When running on Yarn, each application has multiple attempts, so `[app-id]` is actually -`[app-id]/[attempt-id]` in all cases. - -These endpoints have been strongly versioned to make it easier to develop applications on top. - In particular, Spark guarantees: - -* Endpoints will never be removed from one version -* Individual fields will never be removed for any given endpoint -* New endpoints may be added -* New fields may be added to existing endpoints -* New versions of the api may be added in the future at a separate endpoint (eg., `json/v2`). New versions are *not* required to be backwards compatible. -* Api versions may be dropped, but only after at least one minor release of co-existing with a new api version - -Note that even when examining the UI of a running applications, the `applications/[app-id]` portion is -still required, though there is only one application available. Eg. to see the list of jobs for the -running app, you would go to `http://localhost:4040/json/v1/applications/[app-id]/jobs`. This is to -keep the paths consistent in both modes. - # Metrics Spark has a configurable metrics system based on the -- cgit v1.2.3