diff options
author | Matei Zaharia <matei.zaharia@gmail.com> | 2013-09-08 21:39:12 -0700 |
---|---|---|
committer | Matei Zaharia <matei.zaharia@gmail.com> | 2013-09-08 21:39:12 -0700 |
commit | 7d3204b0568e802ba15eb482c1af4c37ebf07022 (patch) | |
tree | 88d167fedb22674d2e61c6aaed0023260a522705 /docs/monitoring.md | |
parent | f1f83712f4e166894e8de855545fa3f4309648f7 (diff) | |
parent | b458854977c437e85fd89056e5d40383c8fa962e (diff) | |
download | spark-7d3204b0568e802ba15eb482c1af4c37ebf07022.tar.gz spark-7d3204b0568e802ba15eb482c1af4c37ebf07022.tar.bz2 spark-7d3204b0568e802ba15eb482c1af4c37ebf07022.zip |
Merge pull request #905 from mateiz/docs2
Job scheduling and cluster mode docs
Diffstat (limited to 'docs/monitoring.md')
-rw-r--r-- | docs/monitoring.md | 30 |
1 files changed, 21 insertions, 9 deletions
diff --git a/docs/monitoring.md b/docs/monitoring.md index 4c4f174503..0e3606f71a 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -3,19 +3,30 @@ layout: global title: Monitoring and Instrumentation --- -There are several ways to monitor the progress of Spark jobs. +There are several ways to monitor Spark applications. # Web Interfaces -When a SparkContext is initialized, it launches a web server (by default at port 3030) which -displays useful information. This includes a list of active and completed scheduler stages, -a summary of RDD blocks and partitions, and environmental information. If multiple SparkContexts -are running on the same host, they will bind to succesive ports beginning with 3030 (3031, 3032, -etc). -Spark's Standlone Mode scheduler also has its own -[web interface](spark-standalone.html#monitoring-and-logging). +Every SparkContext launches a web UI, by default on port 3030, that +displays useful information about the application. This includes: + +* A list of scheduler stages and tasks +* A summary of RDD sizes and memory usage +* Information about the running executors +* Environmental information. + +You can access this interface by simply opening `http://<driver-node>:3030` in a web browser. +If multiple SparkContexts are running on the same host, they will bind to succesive ports +beginning with 3030 (3031, 3032, etc). + +Spark's Standlone Mode cluster manager also has its own +[web UI](spark-standalone.html#monitoring-and-logging). + +Note that in both of these UIs, the tables are sortable by clicking their headers, +making it easy to identify slow tasks, data skew, etc. + +# Metrics -# Spark Metrics Spark has a configurable metrics system based on the [Coda Hale Metrics Library](http://metrics.codahale.com/). This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV @@ -44,6 +55,7 @@ The syntax of the metrics configuration file is defined in an example configurat `$SPARK_HOME/conf/metrics.conf.template`. # Advanced Instrumentation + Several external tools can be used to help profile the performance of Spark jobs: * Cluster-wide monitoring tools, such as [Ganglia](http://ganglia.sourceforge.net/), can provide |