From 35ed09f1d130c16d79b4840f7f87827359c7cb10 Mon Sep 17 00:00:00 2001 From: Jey Kottalam Date: Wed, 4 Sep 2013 11:52:16 -0700 Subject: Clarify YARN example --- docs/running-on-yarn.md | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) (limited to 'docs') diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 93421efcbc..c611db0af4 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -42,7 +42,7 @@ This would be used to connect to the cluster, write to the dfs and submit jobs t The command to launch the YARN Client is as follows: - SPARK_JAR= ./spark-class org.apache.spark.deploy.yarn.Client \ + SPARK_JAR= ./spark-class org.apache.spark.deploy.yarn.Client \ --jar \ --class \ --args \ @@ -54,14 +54,27 @@ The command to launch the YARN Client is as follows: For example: - SPARK_JAR=./yarn/target/spark-yarn-assembly-{{site.SPARK_VERSION}}.jar ./spark-class org.apache.spark.deploy.yarn.Client \ - --jar examples/target/scala-{{site.SCALA_VERSION}}/spark-examples_{{site.SCALA_VERSION}}-{{site.SPARK_VERSION}}.jar \ - --class org.apache.spark.examples.SparkPi \ - --args yarn-standalone \ - --num-workers 3 \ - --master-memory 4g \ - --worker-memory 2g \ - --worker-cores 1 + # Build the Spark assembly JAR and the Spark examples JAR + $ SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true ./sbt/sbt assembly + + # Configure logging + $ cp conf/log4j.properties.template conf/log4j.properties + + # Submit Spark's ApplicationMaster to YARN's ResourceManager, and instruct Spark to run the SparkPi example + $ SPARK_JAR=./assembly/target/scala-{{site.SCALA_VERSION}}/spark-assembly-{{site.SPARK_VERSION}}-hadoop2.0.5-alpha.jar \ + ./spark-class org.apache.spark.deploy.yarn.Client \ + --jar examples/target/scala-{{site.SCALA_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \ + --class org.apache.spark.examples.SparkPi \ + --args yarn-standalone \ + --num-workers 3 \ + --master-memory 4g \ + --worker-memory 2g \ + --worker-cores 1 + + # Examine the output (replace $YARN_APP_ID in the following with the "application identifier" output by the previous command) + # (Note: YARN_APP_LOGS_DIR is usually /tmp/logs or $HADOOP_HOME/logs/userlogs depending on the Hadoop version.) + $ cat $YARN_APP_LOGS_DIR/$YARN_APP_ID/container*_000001/stdout + Pi is roughly 3.13794 The above starts a YARN Client programs which periodically polls the Application Master for status updates and displays them in the console. The client will exit once your application has finished running. -- cgit v1.2.3 From e653a9d8914059fc8430f1d0d4ee9296d8ed9651 Mon Sep 17 00:00:00 2001 From: Patrick Wendell Date: Fri, 6 Sep 2013 12:15:49 -0700 Subject: Provide docs to describe running on CDH/HDP cluster. This doc consolidates information relevant to CDH/HDP users in a single place. --- docs/_layouts/global.html | 1 + docs/cdh-hdp.md | 66 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+) create mode 100644 docs/cdh-hdp.md (limited to 'docs') diff --git a/docs/_layouts/global.html b/docs/_layouts/global.html index 84749fda4e..3a3b8dce37 100755 --- a/docs/_layouts/global.html +++ b/docs/_layouts/global.html @@ -98,6 +98,7 @@
  • Configuration
  • Tuning Guide
    • +
  • Running with CDH/HDP
  • Hardware Provisioning
  • Building Spark with Maven
  • Contributing to Spark
    • diff --git a/docs/cdh-hdp.md b/docs/cdh-hdp.md new file mode 100644 index 0000000000..a4407a135d --- /dev/null +++ b/docs/cdh-hdp.md @@ -0,0 +1,66 @@ +--- +layout: global +title: Running with Cloudera and HortonWorks Distributions +--- + +Spark is fully compatible with all versions of Cloudera's Distribution Including Hadoop (CDH) and +the Hortonworks Data Platform (HDP). There are a few things to keep in mind when running against +these distributions: + +# Compile-time Hadoop Version +When compiling Spark, you'll need to +[set the HADOOP_VERSION flag](http://localhost:4000/index.html#a-note-about-hadoop-versions). +The table below lists the corresponding HADOOP_VERSION for each CDH/HDP release. _Note_ that +some Hadoop releases are binary compatible across client versions. This means the pre-built Spark +distribution may "just work" without you needing to compile. That said, we recommend compiling with +the _exact_ Hadoop version you are running to avoid any compatibility errors. + + + + + + +
    +

    CDH Releases

    + + + + + + + +
    VersionHADOOP_VERSION
    CDH 4.X.X (YARN mode)2.0.0-chd4.X.X
    CDH 4.X.X2.0.0-mr1-chd4.X.X
    CDH 3u60.20.2-cdh3u6
    CDH 3u50.20.2-cdh3u5
    CDH 3u40.20.2-cdh3u4
    +     		+

    HDP Releases

    + + + + + + +
    VersionHADOOP_VERSION
    HDP 1.31.2.0
    HDP 1.21.1.2
    HDP 1.11.0.3
    HDP 1.01.0.3
    +
    + +# Where to Run Spark +As described in the [Hardware Provisioning](hardware-provisioning.html#storage-systems) guide, +Spark can run in a variety of deployment modes: + +* Using dedicated set of Spark nodes in your cluster. These nodes should be co-located with your + Hadoop installation. +* Running on the same nodes as an existing Hadoop installation, with a fixed amount memory and + cores dedicated to Spark on each node. +* Run Spark alongside Hadoop using a cluster resource manager, such as YARN or Mesos. + +These options are identical for those using CDH and HDP. Note that if you are running a YARN +cluster, you may still choose to run Spark on dedicated nodes. In this case, you should use +the `mr1` versions of HADOOP_HOME when compiling, not the YARN versions. + +# Inheriting Cluster Configuration +If you plan to read and write from HDFS using Spark, it is good to include copies of two relevant +Hadoop configuration files in your $SPARK_HOME/conf directory. These are `hdfs-site.xml`, which +provides default behaviors for the HDFS client, and `core-site.xml`, which sets the default +filesystem name. The location of these configuration files varies across CDH and HDP versions, but +a common location is inside of `/etc/hadoop/conf`. Some tools, such as Cloudera Manager, create +configurations on-the-fly, but offer a mechanisms to download copies of them. + +If you can locate these files, copy them into $SPARK_HOME/conf/. -- cgit v1.2.3 From a2a0cf9d681dd01416237bf81ea538e644d65f5d Mon Sep 17 00:00:00 2001 From: Patrick Wendell Date: Thu, 5 Sep 2013 23:33:44 -0700 Subject: Docs describing Spark monitoring and instrumentation --- docs/_layouts/global.html | 1 + docs/monitoring.md | 49 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+) create mode 100644 docs/monitoring.md (limited to 'docs') diff --git a/docs/_layouts/global.html b/docs/_layouts/global.html index 84749fda4e..6138a8f628 100755 --- a/docs/_layouts/global.html +++ b/docs/_layouts/global.html @@ -97,6 +97,7 @@ More
    • Configuration
      • +
    • Monitoring
    • Tuning Guide
    • Hardware Provisioning
    • Building Spark with Maven
      • diff --git a/docs/monitoring.md b/docs/monitoring.md new file mode 100644 index 0000000000..0ec987107c --- /dev/null +++ b/docs/monitoring.md @@ -0,0 +1,49 @@ +--- +layout: global +title: Monitoring and Instrumentation +--- + +There are several ways to monitor the progress of Spark jobs. + +# Web Interfaces +When a SparkContext is initialized, it launches a web server (by default at port 3030) which +displays useful information. This includes a list of active and completed scheduler stages, +a summary of RDD blocks and partitions, and environmental information. If multiple SparkContexts +are running on the same host, they will bind to succesive ports beginning with 3030 (3031, 3032, +etc). + +Spark's Standlone Mode scheduler also has its own +[web interface](spark-standalone.html#monitoring-and-logging). + +# Spark Metrics +Spark has a configurable metrics system based on the +[Coda Hale Metrics Library](http://metrics.codahale.com/). +This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV +files. The metrics system is configured via a configuration file that Spark expects to be present +at `$SPARK_HOME/conf/metrics.conf`. A custom file location can be specified via the +`spark.metrics.conf` Java system property. Spark's metrics are decoupled into different +_instances_ corresponding to Spark components. Within each instance, you can configure a +set of sinks to which metrics are reported. The following instances are currently supported: + +* `master`: The Spark standalone master process. +* `applications`: A component within the master which reports on various applications. +* `worker`: A Spark standalone worker process. +* `executor`: A Spark executor. +* `driver`: The Spark driver process (the process in which your SparkContext is created). + +The syntax of the metrics configuration file is defined in an example configuration file, +`$SPARK_HOME/conf/metrics.conf.template`. + +# Advanced Instrumentation +Several external tools can be used to help profile the performance of Spark jobs: + +* Cluster-wide monitoring tools, such as [Ganglia](http://ganglia.sourceforge.net/), can provide +insight into overall cluster utilization and resource bottlenecks. For instance, a Ganglia +dashboard can quickly reveal whether a particular workload is disk bound, network bound, or +CPU bound. +* OS profiling tools such as [dstat](http://dag.wieers.com/home-made/dstat/), +[iostat](http://linux.die.net/man/1/iostat), and [iotop](http://linux.die.net/man/1/iotop) +can provide fine-grained profiling on individual nodes. +* JVM utilities such as `jstack` for providing stack traces, `jmap` for creating heap-dumps, +`jstat` for reporting time-series statistics and `jconsole` for visually exploring various JVM +properties are useful for those comfortable with JVM internals. -- cgit v1.2.3 From 76d5d2d3c5881782100571fa2976017fa8df4f57 Mon Sep 17 00:00:00 2001 From: Evan Chan Date: Fri, 6 Sep 2013 13:53:00 -0700 Subject: Add notes about starting spark-shell --- docs/spark-standalone.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) (limited to 'docs') diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index 994a96f2c9..7d4bb1d8be 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -22,7 +22,7 @@ Similarly, you can start one or more workers and connect them to the master via: Once you have started a worker, look at the master's web UI ([http://localhost:8080](http://localhost:8080) by default). You should see the new node listed there, along with its number of CPUs and memory (minus one gigabyte left for the OS). -Finally, the following configuration options can be passed to the master and worker: +Finally, the following configuration options can be passed to the master and worker: @@ -134,6 +134,10 @@ To run an interactive Spark shell against the cluster, run the following command MASTER=spark://IP:PORT ./spark-shell +Note that if you are running spark-shell from one of the spark cluster machines, the `spark-shell` script will +automatically set MASTER from the `SPARK_MASTER_IP` and `SPARK_MASTER_PORT` variables in `conf/spark-env.sh`. + +You can also pass an option `-c ` to control the number of cores that spark-shell uses on the cluster. # Job Scheduling -- cgit v1.2.3 From 5a18b854a704fc37dc268f9183552da8655d5b1d Mon Sep 17 00:00:00 2001 From: Evan Chan Date: Fri, 6 Sep 2013 13:59:43 -0700 Subject: Easier way to start the master --- docs/spark-standalone.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs') diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index 7d4bb1d8be..c54de082f9 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -9,7 +9,7 @@ In addition to running on the Mesos or YARN cluster managers, Spark also provide You can start a standalone master server by executing: - ./spark-class org.apache.spark.deploy.master.Master + ./bin/start-master.sh Once started, the master will print out a `spark://HOST:PORT` URL for itself, which you can use to connect workers to it, or pass as the "master" argument to `SparkContext`. You can also find this URL on -- cgit v1.2.3 From 88d53f0dff133920fe14e40a2c4e36dd1c241ec6 Mon Sep 17 00:00:00 2001 From: Evan Chan Date: Fri, 6 Sep 2013 14:03:44 -0700 Subject: "launch" scripts is more accurate terminology --- docs/spark-standalone.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'docs') diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index c54de082f9..d81b4cd0eb 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -3,7 +3,7 @@ layout: global title: Spark Standalone Mode --- -In addition to running on the Mesos or YARN cluster managers, Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use our provided [deploy scripts](#cluster-launch-scripts). It is also possible to run these daemons on a single machine for testing. +In addition to running on the Mesos or YARN cluster managers, Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use our provided [launch scripts](#cluster-launch-scripts). It is also possible to run these daemons on a single machine for testing. # Starting a Cluster Manually @@ -55,7 +55,7 @@ Finally, the following configuration options can be passed to the master and wor # Cluster Launch Scripts -To launch a Spark standalone cluster with the deploy scripts, you need to create a file called `conf/slaves` in your Spark directory, which should contain the hostnames of all the machines where you would like to start Spark workers, one per line. The master machine must be able to access each of the slave machines via password-less `ssh` (using a private key). For testing, you can just put `localhost` in this file. +To launch a Spark standalone cluster with the launch scripts, you need to create a file called `conf/slaves` in your Spark directory, which should contain the hostnames of all the machines where you would like to start Spark workers, one per line. The master machine must be able to access each of the slave machines via password-less `ssh` (using a private key). For testing, you can just put `localhost` in this file. Once you've set up this file, you can launch or stop your cluster with the following shell scripts, based on Hadoop's deploy scripts, and available in `SPARK_HOME/bin`: -- cgit v1.2.3 From ff1dbf210691988cbe8b09aafa37815060fdd7ac Mon Sep 17 00:00:00 2001 From: Evan Chan Date: Fri, 6 Sep 2013 14:20:44 -0700 Subject: Add references to make-distribution.sh --- docs/index.md | 8 ++++++++ docs/spark-standalone.md | 11 +++++++++++ 2 files changed, 19 insertions(+) (limited to 'docs') diff --git a/docs/index.md b/docs/index.md index 7d73929940..ee82c207d7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -21,6 +21,9 @@ Spark uses [Simple Build Tool](http://www.scala-sbt.org), which is bundled with For its Scala API, Spark {{site.SPARK_VERSION}} depends on Scala {{site.SCALA_VERSION}}. If you write applications in Scala, you will need to use this same version of Scala in your own program -- newer major versions may not work. You can get the right version of Scala from [scala-lang.org](http://www.scala-lang.org/download/). +Note: if you are building a binary distribution using `./make-distribution.sh`, you will not need to run +`sbt/sbt assembly`. + # Testing the Build Spark comes with several sample programs in the `examples` directory. @@ -46,6 +49,11 @@ Spark supports several options for deployment: * [Apache Mesos](running-on-mesos.html) * [Hadoop YARN](running-on-yarn.html) +There is a script, `./make-distribution.sh`, which will create a binary distribution of Spark for deployment +to any machine with only the Java runtime as a necessary dependency. +Running the script creates a distribution directory in `dist/`, or the `-tgz` option to create a .tgz file. +Check the script for additional options. + # A Note About Hadoop Versions Spark uses the Hadoop-client library to talk to HDFS and other Hadoop-supported diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index d81b4cd0eb..30641bd777 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -5,6 +5,17 @@ title: Spark Standalone Mode In addition to running on the Mesos or YARN cluster managers, Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use our provided [launch scripts](#cluster-launch-scripts). It is also possible to run these daemons on a single machine for testing. +# Deploying Spark Standalone to a Cluster + +The easiest way to deploy Spark is by running the `./make-distribution.sh` script to create a binary distribution. +This distribution can be deployed to any machine with the Java runtime installed; there is no need to install Scala. + +The recommended procedure is to deploy and start the master on one node first, get the master spark URL, +then modify `conf/spark-env.sh` in the `dist/` directory before deploying to all the other nodes. + +It is also possible to deploy the source directory once you have built it with `sbt assembly`. Scala 2.9.3 +will need to be deployed on all the machines as well, and SCALA_HOME will need to point to the Scala installation. + # Starting a Cluster Manually You can start a standalone master server by executing: -- cgit v1.2.3 From be1ee28ca630e663f54f0ca043e7f1877ccb3da8 Mon Sep 17 00:00:00 2001 From: Evan Chan Date: Sat, 7 Sep 2013 08:56:24 -0700 Subject: CR feedback from Matei --- docs/index.md | 3 --- docs/spark-standalone.md | 5 +---- 2 files changed, 1 insertion(+), 7 deletions(-) (limited to 'docs') diff --git a/docs/index.md b/docs/index.md index ee82c207d7..d3aacc629f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -21,9 +21,6 @@ Spark uses [Simple Build Tool](http://www.scala-sbt.org), which is bundled with For its Scala API, Spark {{site.SPARK_VERSION}} depends on Scala {{site.SCALA_VERSION}}. If you write applications in Scala, you will need to use this same version of Scala in your own program -- newer major versions may not work. You can get the right version of Scala from [scala-lang.org](http://www.scala-lang.org/download/). -Note: if you are building a binary distribution using `./make-distribution.sh`, you will not need to run -`sbt/sbt assembly`. - # Testing the Build Spark comes with several sample programs in the `examples` directory. diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index 30641bd777..69e1291580 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -5,7 +5,7 @@ title: Spark Standalone Mode In addition to running on the Mesos or YARN cluster managers, Spark also provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use our provided [launch scripts](#cluster-launch-scripts). It is also possible to run these daemons on a single machine for testing. -# Deploying Spark Standalone to a Cluster +# Installing Spark Standalone to a Cluster The easiest way to deploy Spark is by running the `./make-distribution.sh` script to create a binary distribution. This distribution can be deployed to any machine with the Java runtime installed; there is no need to install Scala. @@ -13,9 +13,6 @@ This distribution can be deployed to any machine with the Java runtime installed The recommended procedure is to deploy and start the master on one node first, get the master spark URL, then modify `conf/spark-env.sh` in the `dist/` directory before deploying to all the other nodes. -It is also possible to deploy the source directory once you have built it with `sbt assembly`. Scala 2.9.3 -will need to be deployed on all the machines as well, and SCALA_HOME will need to point to the Scala installation. - # Starting a Cluster Manually You can start a standalone master server by executing: -- cgit v1.2.3 From 61c4762d454c7b62bdccd33a5aab5c1f7e8809c5 Mon Sep 17 00:00:00 2001 From: Patrick Wendell Date: Sat, 7 Sep 2013 11:51:40 -0700 Subject: Changes based on feedback --- docs/cdh-hdp.md | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) (limited to 'docs') diff --git a/docs/cdh-hdp.md b/docs/cdh-hdp.md index a4407a135d..679fb1100d 100644 --- a/docs/cdh-hdp.md +++ b/docs/cdh-hdp.md @@ -3,14 +3,17 @@ layout: global title: Running with Cloudera and HortonWorks Distributions --- -Spark is fully compatible with all versions of Cloudera's Distribution Including Hadoop (CDH) and -the Hortonworks Data Platform (HDP). There are a few things to keep in mind when running against +Spark can run against all versions of Cloudera's Distribution Including Hadoop (CDH) and +the Hortonworks Data Platform (HDP). There are a few things to keep in mind when using Spark with these distributions: # Compile-time Hadoop Version When compiling Spark, you'll need to -[set the HADOOP_VERSION flag](http://localhost:4000/index.html#a-note-about-hadoop-versions). -The table below lists the corresponding HADOOP_VERSION for each CDH/HDP release. _Note_ that +[set the HADOOP_VERSION flag](http://localhost:4000/index.html#a-note-about-hadoop-versions): + + HADOOP_VERSION=1.0.4 sbt/sbt assembly + +The table below lists the corresponding HADOOP_VERSION for each CDH/HDP release. Note that some Hadoop releases are binary compatible across client versions. This means the pre-built Spark distribution may "just work" without you needing to compile. That said, we recommend compiling with the _exact_ Hadoop version you are running to avoid any compatibility errors. @@ -51,16 +54,25 @@ Spark can run in a variety of deployment modes: cores dedicated to Spark on each node. * Run Spark alongside Hadoop using a cluster resource manager, such as YARN or Mesos. -These options are identical for those using CDH and HDP. Note that if you are running a YARN -cluster, you may still choose to run Spark on dedicated nodes. In this case, you should use -the `mr1` versions of HADOOP_HOME when compiling, not the YARN versions. +These options are identical for those using CDH and HDP. Note that if you have a YARN cluster, +but still prefer to run Spark on a dedicated set of nodes rather than scheduling through YARN, +use `mr1` versions of HADOOP_HOME when compiling. # Inheriting Cluster Configuration -If you plan to read and write from HDFS using Spark, it is good to include copies of two relevant -Hadoop configuration files in your $SPARK_HOME/conf directory. These are `hdfs-site.xml`, which -provides default behaviors for the HDFS client, and `core-site.xml`, which sets the default -filesystem name. The location of these configuration files varies across CDH and HDP versions, but +If you plan to read and write from HDFS using Spark, there are two Hadoop configuration files that +should be included on Spark's classpath: + +* `hdfs-site.xml`, which provides default behaviors for the HDFS client. +* `core-site.xml`, which sets the default filesystem name. + +The location of these configuration files varies across CDH and HDP versions, but a common location is inside of `/etc/hadoop/conf`. Some tools, such as Cloudera Manager, create configurations on-the-fly, but offer a mechanisms to download copies of them. -If you can locate these files, copy them into $SPARK_HOME/conf/. +There are a few ways to make these files visible to Spark: + +* You can copy these files into `$SPARK_HOME/conf` and they will be included in Spark's +classpath automatically. +* If you are running Spark on the same nodes as Hadoop _and_ your distribution includes both +`hdfs-site.xml` and `core-site.xml` in the same directory, you can set `HADOOP_CONF_DIR` +in `$SPARK_HOME/spark-env.sh` to that directory. -- cgit v1.2.3 From 22b982d2bc76197c85ed50558a0fc676dffcc5ef Mon Sep 17 00:00:00 2001 From: Patrick Wendell Date: Sat, 7 Sep 2013 14:37:54 -0700 Subject: File rename --- docs/_layouts/global.html | 2 +- docs/cdh-hdp.md | 78 -------------------------------- docs/hadoop-third-party-distributions.md | 76 +++++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+), 79 deletions(-) delete mode 100644 docs/cdh-hdp.md create mode 100644 docs/hadoop-third-party-distributions.md (limited to 'docs') diff --git a/docs/_layouts/global.html b/docs/_layouts/global.html index 3a3b8dce37..2f6bdcabe8 100755 --- a/docs/_layouts/global.html +++ b/docs/_layouts/global.html @@ -98,7 +98,7 @@
      • Configuration
      • Tuning Guide
        • -
      • Running with CDH/HDP
        • +
      • Running with CDH/HDP
      • Hardware Provisioning
      • Building Spark with Maven
      • Contributing to Spark
        • diff --git a/docs/cdh-hdp.md b/docs/cdh-hdp.md deleted file mode 100644 index 679fb1100d..0000000000 --- a/docs/cdh-hdp.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -layout: global -title: Running with Cloudera and HortonWorks Distributions ---- - -Spark can run against all versions of Cloudera's Distribution Including Hadoop (CDH) and -the Hortonworks Data Platform (HDP). There are a few things to keep in mind when using Spark with -these distributions: - -# Compile-time Hadoop Version -When compiling Spark, you'll need to -[set the HADOOP_VERSION flag](http://localhost:4000/index.html#a-note-about-hadoop-versions): - - HADOOP_VERSION=1.0.4 sbt/sbt assembly - -The table below lists the corresponding HADOOP_VERSION for each CDH/HDP release. Note that -some Hadoop releases are binary compatible across client versions. This means the pre-built Spark -distribution may "just work" without you needing to compile. That said, we recommend compiling with -the _exact_ Hadoop version you are running to avoid any compatibility errors. - -
      ArgumentMeaning
      - - - - -
      -

      CDH Releases

      - - - - - - - -
      VersionHADOOP_VERSION
      CDH 4.X.X (YARN mode)2.0.0-chd4.X.X
      CDH 4.X.X2.0.0-mr1-chd4.X.X
      CDH 3u60.20.2-cdh3u6
      CDH 3u50.20.2-cdh3u5
      CDH 3u40.20.2-cdh3u4
      -       		-

      HDP Releases

      - - - - - - -
      VersionHADOOP_VERSION
      HDP 1.31.2.0
      HDP 1.21.1.2
      HDP 1.11.0.3
      HDP 1.01.0.3
      -
      - -# Where to Run Spark -As described in the [Hardware Provisioning](hardware-provisioning.html#storage-systems) guide, -Spark can run in a variety of deployment modes: - -* Using dedicated set of Spark nodes in your cluster. These nodes should be co-located with your - Hadoop installation. -* Running on the same nodes as an existing Hadoop installation, with a fixed amount memory and - cores dedicated to Spark on each node. -* Run Spark alongside Hadoop using a cluster resource manager, such as YARN or Mesos. - -These options are identical for those using CDH and HDP. Note that if you have a YARN cluster, -but still prefer to run Spark on a dedicated set of nodes rather than scheduling through YARN, -use `mr1` versions of HADOOP_HOME when compiling. - -# Inheriting Cluster Configuration -If you plan to read and write from HDFS using Spark, there are two Hadoop configuration files that -should be included on Spark's classpath: - -* `hdfs-site.xml`, which provides default behaviors for the HDFS client. -* `core-site.xml`, which sets the default filesystem name. - -The location of these configuration files varies across CDH and HDP versions, but -a common location is inside of `/etc/hadoop/conf`. Some tools, such as Cloudera Manager, create -configurations on-the-fly, but offer a mechanisms to download copies of them. - -There are a few ways to make these files visible to Spark: - -* You can copy these files into `$SPARK_HOME/conf` and they will be included in Spark's -classpath automatically. -* If you are running Spark on the same nodes as Hadoop _and_ your distribution includes both -`hdfs-site.xml` and `core-site.xml` in the same directory, you can set `HADOOP_CONF_DIR` -in `$SPARK_HOME/spark-env.sh` to that directory. diff --git a/docs/hadoop-third-party-distributions.md b/docs/hadoop-third-party-distributions.md new file mode 100644 index 0000000000..9f4f354525 --- /dev/null +++ b/docs/hadoop-third-party-distributions.md @@ -0,0 +1,76 @@ +--- +layout: global +title: Running with Cloudera and HortonWorks Distributions +--- + +Spark can run against all versions of Cloudera's Distribution Including Hadoop (CDH) and +the Hortonworks Data Platform (HDP). There are a few things to keep in mind when using Spark with +these distributions: + +# Compile-time Hadoop Version +When compiling Spark, you'll need to +[set the HADOOP_VERSION flag](http://localhost:4000/index.html#a-note-about-hadoop-versions): + + HADOOP_VERSION=1.0.4 sbt/sbt assembly + +The table below lists the corresponding HADOOP_VERSION for each CDH/HDP release. Note that +some Hadoop releases are binary compatible across client versions. This means the pre-built Spark +distribution may "just work" without you needing to compile. That said, we recommend compiling with +the _exact_ Hadoop version you are running to avoid any compatibility errors. + + + + + + +
      +

      CDH Releases

      + + + + + + + +
      VersionHADOOP_VERSION
      CDH 4.X.X (YARN mode)2.0.0-chd4.X.X
      CDH 4.X.X2.0.0-mr1-chd4.X.X
      CDH 3u60.20.2-cdh3u6
      CDH 3u50.20.2-cdh3u5
      CDH 3u40.20.2-cdh3u4
      +       		+

      HDP Releases

      + + + + + + +
      VersionHADOOP_VERSION
      HDP 1.31.2.0
      HDP 1.21.1.2
      HDP 1.11.0.3
      HDP 1.01.0.3
      +
      + +# Where to Run Spark +As described in the [Hardware Provisioning](hardware-provisioning.html#storage-systems) guide, +Spark can run in a variety of deployment modes: + +* Using dedicated set of Spark nodes in your cluster. These nodes should be co-located with your + Hadoop installation. +* Running on the same nodes as an existing Hadoop installation, with a fixed amount memory and + cores dedicated to Spark on each node. +* Run Spark alongside Hadoop using a cluster resource manager, such as YARN or Mesos. + +These options are identical for those using CDH and HDP. + +# Inheriting Cluster Configuration +If you plan to read and write from HDFS using Spark, there are two Hadoop configuration files that +should be included on Spark's classpath: + +* `hdfs-site.xml`, which provides default behaviors for the HDFS client. +* `core-site.xml`, which sets the default filesystem name. + +The location of these configuration files varies across CDH and HDP versions, but +a common location is inside of `/etc/hadoop/conf`. Some tools, such as Cloudera Manager, create +configurations on-the-fly, but offer a mechanisms to download copies of them. + +There are a few ways to make these files visible to Spark: + +* You can copy these files into `$SPARK_HOME/conf` and they will be included in Spark's +classpath automatically. +* If you are running Spark on the same nodes as Hadoop _and_ your distribution includes both +`hdfs-site.xml` and `core-site.xml` in the same directory, you can set `HADOOP_CONF_DIR` +in `$SPARK_HOME/spark-env.sh` to that directory. -- cgit v1.2.3 From c190b48bf5073b3349b5060c324c890d95bc4260 Mon Sep 17 00:00:00 2001 From: Patrick Wendell Date: Sun, 8 Sep 2013 10:47:45 -0700 Subject: Adding more docs and some code cleanup --- conf/metrics.properties.template | 36 +++++++++++++++++++++- .../org/apache/spark/metrics/MetricsConfig.scala | 7 ++--- .../apache/spark/metrics/sink/MetricsServlet.scala | 11 ++++--- .../apache/spark/metrics/MetricsConfigSuite.scala | 19 +++++------- docs/monitoring.md | 9 ++++++ 5 files changed, 62 insertions(+), 20 deletions(-) (limited to 'docs') diff --git a/conf/metrics.properties.template b/conf/metrics.properties.template index 6c36f3cca4..f0e033eb98 100644 --- a/conf/metrics.properties.template +++ b/conf/metrics.properties.template @@ -31,7 +31,7 @@ # 1. To add a new sink, set the "class" option to a fully qualified class # name (see examples below). # 2. Some sinks involve a polling period. The minimum allowed polling period -# is 1 second. +# is 1 second. # 3. Wild card properties can be overridden by more specific properties. # For example, master.sink.console.period takes precedence over # *.sink.console.period. @@ -47,6 +47,40 @@ # instance master and applications. MetricsServlet may not be configured by self. # +## List of available sinks and their properties. + +# org.apache.spark.metrics.sink.ConsoleSink +# Name: Default: Description: +# period 10 Poll period +# unit seconds Units of poll period + +# org.apache.spark.metrics.sink.CSVSink +# Name: Default: Description: +# period 10 Poll period +# unit seconds Units of poll period +# directory /tmp Where to store CSV files + +# org.apache.spark.metrics.sink.GangliaSink +# Name: Default: Description: +# host NONE Hostname or multicast group of Ganglia server +# port NONE Port of Ganglia server(s) +# period 10 Poll period +# unit seconds Units of poll period +# ttl 1 TTL of messages sent by Ganglia +# mode multicast Ganglia network mode ('unicast' or 'mulitcast') + +# org.apache.spark.metrics.sink.JmxSink + +# org.apache.spark.metrics.sink.MetricsServlet +# Name: Default: Description: +# path VARIES* Path prefix from the web server root +# sample false Whether to show entire set of samples for histograms ('false' or 'true') +# +# * Default path is /metrics/json for all instances except the master. The master has two paths: +# /metrics/aplications/json # App information +# /metrics/master/json # Master information + +## Examples # Enable JmxSink for all instances by class name #*.sink.jmx.class=spark.metrics.sink.JmxSink diff --git a/core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala b/core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala index 0f9c4e00b1..caab748d60 100644 --- a/core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala +++ b/core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala @@ -37,10 +37,9 @@ private[spark] class MetricsConfig(val configFile: Option[String]) extends Loggi private def setDefaultProperties(prop: Properties) { prop.setProperty("*.sink.servlet.class", "org.apache.spark.metrics.sink.MetricsServlet") - prop.setProperty("*.sink.servlet.uri", "/metrics/json") - prop.setProperty("*.sink.servlet.sample", "false") - prop.setProperty("master.sink.servlet.uri", "/metrics/master/json") - prop.setProperty("applications.sink.servlet.uri", "/metrics/applications/json") + prop.setProperty("*.sink.servlet.path", "/metrics/json") + prop.setProperty("master.sink.servlet.path", "/metrics/master/json") + prop.setProperty("applications.sink.servlet.path", "/metrics/applications/json") } def initialize() { diff --git a/core/src/main/scala/org/apache/spark/metrics/sink/MetricsServlet.scala b/core/src/main/scala/org/apache/spark/metrics/sink/MetricsServlet.scala index 4e90dd4323..99357fede6 100644 --- a/core/src/main/scala/org/apache/spark/metrics/sink/MetricsServlet.scala +++ b/core/src/main/scala/org/apache/spark/metrics/sink/MetricsServlet.scala @@ -31,18 +31,21 @@ import org.eclipse.jetty.server.Handler import org.apache.spark.ui.JettyUtils class MetricsServlet(val property: Properties, val registry: MetricRegistry) extends Sink { - val SERVLET_KEY_URI = "uri" + val SERVLET_KEY_PATH = "path" val SERVLET_KEY_SAMPLE = "sample" - val servletURI = property.getProperty(SERVLET_KEY_URI) + val SERVLET_DEFAULT_SAMPLE = false - val servletShowSample = property.getProperty(SERVLET_KEY_SAMPLE).toBoolean + val servletPath = property.getProperty(SERVLET_KEY_PATH) + + val servletShowSample = Option(property.getProperty(SERVLET_KEY_SAMPLE)).map(_.toBoolean) + .getOrElse(SERVLET_DEFAULT_SAMPLE) val mapper = new ObjectMapper().registerModule( new MetricsModule(TimeUnit.SECONDS, TimeUnit.MILLISECONDS, servletShowSample)) def getHandlers = Array[(String, Handler)]( - (servletURI, JettyUtils.createHandler(request => getMetricsSnapshot(request), "text/json")) + (servletPath, JettyUtils.createHandler(request => getMetricsSnapshot(request), "text/json")) ) def getMetricsSnapshot(request: HttpServletRequest): String = { diff --git a/core/src/test/scala/org/apache/spark/metrics/MetricsConfigSuite.scala b/core/src/test/scala/org/apache/spark/metrics/MetricsConfigSuite.scala index 58c94a162d..1a9ce8c607 100644 --- a/core/src/test/scala/org/apache/spark/metrics/MetricsConfigSuite.scala +++ b/core/src/test/scala/org/apache/spark/metrics/MetricsConfigSuite.scala @@ -30,14 +30,13 @@ class MetricsConfigSuite extends FunSuite with BeforeAndAfter { val conf = new MetricsConfig(Option("dummy-file")) conf.initialize() - assert(conf.properties.size() === 5) + assert(conf.properties.size() === 4) assert(conf.properties.getProperty("test-for-dummy") === null) val property = conf.getInstance("random") - assert(property.size() === 3) + assert(property.size() === 2) assert(property.getProperty("sink.servlet.class") === "org.apache.spark.metrics.sink.MetricsServlet") - assert(property.getProperty("sink.servlet.uri") === "/metrics/json") - assert(property.getProperty("sink.servlet.sample") === "false") + assert(property.getProperty("sink.servlet.path") === "/metrics/json") } test("MetricsConfig with properties set") { @@ -45,22 +44,20 @@ class MetricsConfigSuite extends FunSuite with BeforeAndAfter { conf.initialize() val masterProp = conf.getInstance("master") - assert(masterProp.size() === 6) + assert(masterProp.size() === 5) assert(masterProp.getProperty("sink.console.period") === "20") assert(masterProp.getProperty("sink.console.unit") === "minutes") assert(masterProp.getProperty("source.jvm.class") === "org.apache.spark.metrics.source.JvmSource") assert(masterProp.getProperty("sink.servlet.class") === "org.apache.spark.metrics.sink.MetricsServlet") - assert(masterProp.getProperty("sink.servlet.uri") === "/metrics/master/json") - assert(masterProp.getProperty("sink.servlet.sample") === "false") + assert(masterProp.getProperty("sink.servlet.path") === "/metrics/master/json") val workerProp = conf.getInstance("worker") - assert(workerProp.size() === 6) + assert(workerProp.size() === 5) assert(workerProp.getProperty("sink.console.period") === "10") assert(workerProp.getProperty("sink.console.unit") === "seconds") assert(workerProp.getProperty("source.jvm.class") === "org.apache.spark.metrics.source.JvmSource") assert(workerProp.getProperty("sink.servlet.class") === "org.apache.spark.metrics.sink.MetricsServlet") - assert(workerProp.getProperty("sink.servlet.uri") === "/metrics/json") - assert(workerProp.getProperty("sink.servlet.sample") === "false") + assert(workerProp.getProperty("sink.servlet.path") === "/metrics/json") } test("MetricsConfig with subProperties") { @@ -84,6 +81,6 @@ class MetricsConfigSuite extends FunSuite with BeforeAndAfter { assert(consoleProps.size() === 2) val servletProps = sinkProps("servlet") - assert(servletProps.size() === 3) + assert(servletProps.size() === 2) } } diff --git a/docs/monitoring.md b/docs/monitoring.md index 0ec987107c..4c4f174503 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -31,6 +31,15 @@ set of sinks to which metrics are reported. The following instances are currentl * `executor`: A Spark executor. * `driver`: The Spark driver process (the process in which your SparkContext is created). +Each instance can report to zero or more _sinks_. Sinks are contained in the +`org.apache.spark.metrics.sink` package: + +* `ConsoleSink`: Logs metrics information to the console. +* `CSVSink`: Exports metrics data to CSV files at regular intervals. +* `GangliaSink`: Sends metrics to a Ganglia node or multicast group. +* `JmxSink`: Registers metrics for viewing in a JXM console. +* `MetricsServlet`: Adds a servlet within the existing Spark UI to serve metrics data as JSON data. + The syntax of the metrics configuration file is defined in an example configuration file, `$SPARK_HOME/conf/metrics.conf.template`. -- cgit v1.2.3