diff options
author | Patrick Wendell <pwendell@gmail.com> | 2013-09-06 15:25:22 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2013-09-06 15:25:22 -0700 |
commit | 2eebeff5ebe86509adb2a0f18521d1648ea38008 (patch) | |
tree | 0a61eb9189c3d91440708eb2371e6a18c67907a2 | |
parent | 1e15feb5a314e7180328b9208054966e040eb2ad (diff) | |
parent | a2a0cf9d681dd01416237bf81ea538e644d65f5d (diff) | |
download | spark-2eebeff5ebe86509adb2a0f18521d1648ea38008.tar.gz spark-2eebeff5ebe86509adb2a0f18521d1648ea38008.tar.bz2 spark-2eebeff5ebe86509adb2a0f18521d1648ea38008.zip |
Merge pull request #897 from pwendell/master
Docs describing Spark monitoring and instrumentation
-rwxr-xr-x | docs/_layouts/global.html | 1 | ||||
-rw-r--r-- | docs/monitoring.md | 49 |
2 files changed, 50 insertions, 0 deletions
diff --git a/docs/_layouts/global.html b/docs/_layouts/global.html index 84749fda4e..6138a8f628 100755 --- a/docs/_layouts/global.html +++ b/docs/_layouts/global.html @@ -97,6 +97,7 @@ <a href="api.html" class="dropdown-toggle" data-toggle="dropdown">More<b class="caret"></b></a> <ul class="dropdown-menu"> <li><a href="configuration.html">Configuration</a></li> + <li><a href="monitoring.html">Monitoring</a></li> <li><a href="tuning.html">Tuning Guide</a></li> <li><a href="hardware-provisioning.html">Hardware Provisioning</a></li> <li><a href="building-with-maven.html">Building Spark with Maven</a></li> diff --git a/docs/monitoring.md b/docs/monitoring.md new file mode 100644 index 0000000000..0ec987107c --- /dev/null +++ b/docs/monitoring.md @@ -0,0 +1,49 @@ +--- +layout: global +title: Monitoring and Instrumentation +--- + +There are several ways to monitor the progress of Spark jobs. + +# Web Interfaces +When a SparkContext is initialized, it launches a web server (by default at port 3030) which +displays useful information. This includes a list of active and completed scheduler stages, +a summary of RDD blocks and partitions, and environmental information. If multiple SparkContexts +are running on the same host, they will bind to succesive ports beginning with 3030 (3031, 3032, +etc). + +Spark's Standlone Mode scheduler also has its own +[web interface](spark-standalone.html#monitoring-and-logging). + +# Spark Metrics +Spark has a configurable metrics system based on the +[Coda Hale Metrics Library](http://metrics.codahale.com/). +This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV +files. The metrics system is configured via a configuration file that Spark expects to be present +at `$SPARK_HOME/conf/metrics.conf`. A custom file location can be specified via the +`spark.metrics.conf` Java system property. Spark's metrics are decoupled into different +_instances_ corresponding to Spark components. Within each instance, you can configure a +set of sinks to which metrics are reported. The following instances are currently supported: + +* `master`: The Spark standalone master process. +* `applications`: A component within the master which reports on various applications. +* `worker`: A Spark standalone worker process. +* `executor`: A Spark executor. +* `driver`: The Spark driver process (the process in which your SparkContext is created). + +The syntax of the metrics configuration file is defined in an example configuration file, +`$SPARK_HOME/conf/metrics.conf.template`. + +# Advanced Instrumentation +Several external tools can be used to help profile the performance of Spark jobs: + +* Cluster-wide monitoring tools, such as [Ganglia](http://ganglia.sourceforge.net/), can provide +insight into overall cluster utilization and resource bottlenecks. For instance, a Ganglia +dashboard can quickly reveal whether a particular workload is disk bound, network bound, or +CPU bound. +* OS profiling tools such as [dstat](http://dag.wieers.com/home-made/dstat/), +[iostat](http://linux.die.net/man/1/iostat), and [iotop](http://linux.die.net/man/1/iotop) +can provide fine-grained profiling on individual nodes. +* JVM utilities such as `jstack` for providing stack traces, `jmap` for creating heap-dumps, +`jstat` for reporting time-series statistics and `jconsole` for visually exploring various JVM +properties are useful for those comfortable with JVM internals. |