From 98fb69822cf780160bca51abeaab7c82e49fab54 Mon Sep 17 00:00:00 2001 From: Matei Zaharia Date: Fri, 6 Sep 2013 00:29:37 -0400 Subject: Work in progress: - Add job scheduling docs - Rename some fair scheduler properties - Organize intro page better - Link to Apache wiki for "contributing to Spark" --- docs/cluster-overview.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 docs/cluster-overview.md (limited to 'docs/cluster-overview.md') diff --git a/docs/cluster-overview.md b/docs/cluster-overview.md new file mode 100644 index 0000000000..9e781bbf1f --- /dev/null +++ b/docs/cluster-overview.md @@ -0,0 +1,5 @@ +--- +layout: global +title: Cluster Mode Overview +--- + -- cgit v1.2.3 From f261d2a60fe9c0ec81c7a93a24fd79062c31f7ae Mon Sep 17 00:00:00 2001 From: Matei Zaharia Date: Sun, 8 Sep 2013 00:41:18 -0400 Subject: Added cluster overview doc, made logo higher-resolution, and added more details on monitoring --- docs/_layouts/global.html | 3 +- docs/cluster-overview.md | 65 +++++++++++++++++++++++++++++++++++++++++ docs/img/cluster-overview.png | Bin 0 -> 28816 bytes docs/img/cluster-overview.pptx | Bin 0 -> 49698 bytes docs/img/spark-logo-hd.png | Bin 0 -> 13512 bytes docs/index.md | 5 ---- docs/monitoring.md | 30 +++++++++++++------ 7 files changed, 88 insertions(+), 15 deletions(-) create mode 100644 docs/img/cluster-overview.png create mode 100644 docs/img/cluster-overview.pptx create mode 100644 docs/img/spark-logo-hd.png (limited to 'docs/cluster-overview.md') diff --git a/docs/_layouts/global.html b/docs/_layouts/global.html index 5034111ecb..238ad26de0 100755 --- a/docs/_layouts/global.html +++ b/docs/_layouts/global.html @@ -51,7 +51,7 @@
- {{site.SPARK_VERSION_SHORT}} + {{site.SPARK_VERSION_SHORT}}
diff --git a/docs/cluster-overview.md b/docs/cluster-overview.md index 9e781bbf1f..143f93171f 100644 --- a/docs/cluster-overview.md +++ b/docs/cluster-overview.md @@ -3,3 +3,68 @@ layout: global title: Cluster Mode Overview --- +This document gives a short overview of how Spark runs on clusters, to make it easier to understand +the components involved. + +# Components + +Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext +object in your main program (called the _driver program_). +Specifically, to run on a cluster, the SparkContext can connect to several types of _cluster managers_ +(either Spark's own standalone cluster manager or Mesos/YARN), which allocate resources across +applications. Once connected, Spark acquires *executors* on nodes in the cluster, which are +worker processes that run computations and store data for your application. +Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to +the executors. Finally, SparkContext sends *tasks* for the executors to run. + +

+ Spark cluster components +

+ +There are several useful things to note about this architecture: + +1. Each application gets its own executor processes, which stay up for the duration of the whole + application and run tasks in multiple threads. This has the benefit of isolating applications + from each other, on both the scheduling side (each driver schedules its own tasks) and executor + side (tasks from different applications run in different JVMs). However, it also means that + data cannot be shared across different Spark applications (instances of SparkContext) without + writing it to an external storage system. +2. Spark is agnostic to the underlying cluster manager. As long as it can acquire executor + processes, and these communicate with each other, it is relatively easy to run it even on a + cluster manager that also supports other applications (e.g. Mesos/YARN). +3. Because the driver schedules tasks on the cluster, it should be run close to the worker + nodes, preferably on the same local area network. If you'd like to send requests to the + cluster remotely, it's better to open an RPC to the driver and have it submit operations + from nearby than to run a driver far away from the worker nodes. + +# Cluster Manager Types + +The system currently supports three cluster managers: + +* [Standalone](spark-standalone.html) -- a simple cluster manager included with Spark that makes it + easy to set up a cluster. +* [Apache Mesos](running-on-mesos.html) -- a general cluster manager that can also run Hadoop MapReduce + and service applications. +* [Hadoop YARN](running-on-yarn.html) -- the resource manager in Hadoop 2.0. + +In addition, Spark's [EC2 launch scripts](ec2-scripts.html) make it easy to launch a standalone +cluster on Amazon EC2. + +# Shipping Code to the Cluster + +The recommended way to ship your code to the cluster is to pass it through SparkContext's constructor, +which takes a list of JAR files (Java/Scala) or .egg and .zip libraries (Python) to disseminate to +worker nodes. You can also dynamically add new files to be sent to executors with `SparkContext.addJar` +and `addFile`. + +# Monitoring + +Each driver program has a web UI, typically on port 3030, that displays information about running +tasks, executors, and storage usage. Simply go to `http://:3030` in a web browser to +access this UI. The [monitoring guide](monitoring.html) also describes other monitoring options. + +# Job Scheduling + +Spark gives control over resource allocation both _across_ applications (at the level of the cluster +manager) and _within_ applications (if multiple computations are happening on the same SparkContext). +The [job scheduling overview](job-scheduling.html) describes this in more detail. diff --git a/docs/img/cluster-overview.png b/docs/img/cluster-overview.png new file mode 100644 index 0000000000..2a1cf02fcf Binary files /dev/null and b/docs/img/cluster-overview.png differ diff --git a/docs/img/cluster-overview.pptx b/docs/img/cluster-overview.pptx new file mode 100644 index 0000000000..2a61db352d Binary files /dev/null and b/docs/img/cluster-overview.pptx differ diff --git a/docs/img/spark-logo-hd.png b/docs/img/spark-logo-hd.png new file mode 100644 index 0000000000..1381e3004d Binary files /dev/null and b/docs/img/spark-logo-hd.png differ diff --git a/docs/index.md b/docs/index.md index 1814cb19c8..bd386a8a8f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -48,11 +48,6 @@ options for deployment: * [Apache Mesos](running-on-mesos.html) * [Hadoop YARN](running-on-yarn.html) -There is a script, `./make-distribution.sh`, which will create a binary distribution of Spark for deployment -to any machine with only the Java runtime as a necessary dependency. -Running the script creates a distribution directory in `dist/`, or the `-tgz` option to create a .tgz file. -Check the script for additional options. - # A Note About Hadoop Versions Spark uses the Hadoop-client library to talk to HDFS and other Hadoop-supported diff --git a/docs/monitoring.md b/docs/monitoring.md index 0ec987107c..e9832e0466 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -3,19 +3,30 @@ layout: global title: Monitoring and Instrumentation --- -There are several ways to monitor the progress of Spark jobs. +There are several ways to monitor Spark applications. # Web Interfaces -When a SparkContext is initialized, it launches a web server (by default at port 3030) which -displays useful information. This includes a list of active and completed scheduler stages, -a summary of RDD blocks and partitions, and environmental information. If multiple SparkContexts -are running on the same host, they will bind to succesive ports beginning with 3030 (3031, 3032, -etc). -Spark's Standlone Mode scheduler also has its own -[web interface](spark-standalone.html#monitoring-and-logging). +Every SparkContext launches a web UI, by default on port 3030, that +displays useful information about the application. This includes: + +* A list of scheduler stages and tasks +* A summary of RDD sizes and memory usage +* Information about the running executors +* Environmental information. + +You can access this interface by simply opening `http://:3030` in a web browser. +If multiple SparkContexts are running on the same host, they will bind to succesive ports +beginning with 3030 (3031, 3032, etc). + +Spark's Standlone Mode cluster manager also has its own +[web UI](spark-standalone.html#monitoring-and-logging). + +Note that in both of these UIs, the tables are sortable by clicking their headers, +making it easy to identify slow tasks, data skew, etc. + +# Metrics -# Spark Metrics Spark has a configurable metrics system based on the [Coda Hale Metrics Library](http://metrics.codahale.com/). This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV @@ -35,6 +46,7 @@ The syntax of the metrics configuration file is defined in an example configurat `$SPARK_HOME/conf/metrics.conf.template`. # Advanced Instrumentation + Several external tools can be used to help profile the performance of Spark jobs: * Cluster-wide monitoring tools, such as [Ganglia](http://ganglia.sourceforge.net/), can provide -- cgit v1.2.3 From af8ffdb73c28012c9f5cf232ca7d4b4c6763628d Mon Sep 17 00:00:00 2001 From: Matei Zaharia Date: Sun, 8 Sep 2013 13:36:50 -0700 Subject: Review comments --- docs/cluster-overview.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ docs/job-scheduling.md | 2 +- 2 files changed, 48 insertions(+), 1 deletion(-) (limited to 'docs/cluster-overview.md') diff --git a/docs/cluster-overview.md b/docs/cluster-overview.md index 143f93171f..cf6b48c05e 100644 --- a/docs/cluster-overview.md +++ b/docs/cluster-overview.md @@ -68,3 +68,50 @@ access this UI. The [monitoring guide](monitoring.html) also describes other mon Spark gives control over resource allocation both _across_ applications (at the level of the cluster manager) and _within_ applications (if multiple computations are happening on the same SparkContext). The [job scheduling overview](job-scheduling.html) describes this in more detail. + +# Glossary + +The following table summarizes terms you'll see used to refer to cluster concepts: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TermMeaning
ApplicationAny user program invoking Spark
Driver programThe process running the main() function of the application and creating the SparkContext
Cluster managerAn external service for acquiring resources on the cluster (e.g. standalone manager, Mesos, YARN)
Worker nodeAny node that can run application code in the cluster
ExecutorA process launched for an application on a worker node, that runs tasks and keeps data in memory + or disk storage across them. Each application has its own executors.
TaskA unit of work that will be sent to one executor
JobA parallel computation consisting of multiple tasks that gets spawned in response to a Spark action + (e.g. save, collect); you'll see this term used in the driver's logs.
StageEach job gets divided into smaller sets of tasks called stages that depend on each other + (similar to the map and reduce stages in MapReduce); you'll see this term used in the driver's logs.
diff --git a/docs/job-scheduling.md b/docs/job-scheduling.md index 11b733137d..d304c5497b 100644 --- a/docs/job-scheduling.md +++ b/docs/job-scheduling.md @@ -25,7 +25,7 @@ different options to manage allocation, depending on the cluster manager. The simplest option, available on all cluster managers, is _static partitioning_ of resources. With this approach, each application is given a maximum amount of resources it can use, and holds onto them -for its whole duration. This is the only approach available in Spark's [standalone](spark-standalone.html) +for its whole duration. This is the approach used in Spark's [standalone](spark-standalone.html) and [YARN](running-on-yarn.html) modes, as well as the [coarse-grained Mesos mode](running-on-mesos.html#mesos-run-modes). Resource allocation can be configured as follows, based on the cluster type: -- cgit v1.2.3 From b458854977c437e85fd89056e5d40383c8fa962e Mon Sep 17 00:00:00 2001 From: Matei Zaharia Date: Sun, 8 Sep 2013 21:25:49 -0700 Subject: Fix some review comments --- docs/cluster-overview.md | 2 +- docs/quick-start.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) (limited to 'docs/cluster-overview.md') diff --git a/docs/cluster-overview.md b/docs/cluster-overview.md index cf6b48c05e..7025c23657 100644 --- a/docs/cluster-overview.md +++ b/docs/cluster-overview.md @@ -80,7 +80,7 @@ The following table summarizes terms you'll see used to refer to cluster concept Application - Any user program invoking Spark + User program built on Spark. Consists of a driver program and executors on the cluster. Driver program diff --git a/docs/quick-start.md b/docs/quick-start.md index 1b069ce982..8f782db5b8 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -36,7 +36,7 @@ scala> textFile.count() // Number of items in this RDD res0: Long = 74 scala> textFile.first() // First item in this RDD -res1: String = Welcome to the Spark documentation! +res1: String = # Apache Spark {% endhighlight %} Now let's use a transformation. We will use the [`filter`](scala-programming-guide.html#transformations) transformation to return a new RDD with a subset of the items in the file. -- cgit v1.2.3