From e653a9d8914059fc8430f1d0d4ee9296d8ed9651 Mon Sep 17 00:00:00 2001
From: Patrick Wendell <pwendell@gmail.com>
Date: Fri, 6 Sep 2013 12:15:49 -0700
Subject: Provide docs to describe running on CDH/HDP cluster.

This doc consolidates information relevant to CDH/HDP users in a single place.
---
 docs/_layouts/global.html |  1 +
 docs/cdh-hdp.md           | 66 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 67 insertions(+)
 create mode 100644 docs/cdh-hdp.md

diff --git a/docs/_layouts/global.html b/docs/_layouts/global.html
index 84749fda4e..3a3b8dce37 100755
--- a/docs/_layouts/global.html
+++ b/docs/_layouts/global.html
@@ -98,6 +98,7 @@
                             <ul class="dropdown-menu">
                                 <li><a href="configuration.html">Configuration</a></li>
                                 <li><a href="tuning.html">Tuning Guide</a></li>
+                                <li><a href="cdh-hdp.html">Running with CDH/HDP</a></li>
                                 <li><a href="hardware-provisioning.html">Hardware Provisioning</a></li>
                                 <li><a href="building-with-maven.html">Building Spark with Maven</a></li>
                                 <li><a href="contributing-to-spark.html">Contributing to Spark</a></li>
diff --git a/docs/cdh-hdp.md b/docs/cdh-hdp.md
new file mode 100644
index 0000000000..a4407a135d
--- /dev/null
+++ b/docs/cdh-hdp.md
@@ -0,0 +1,66 @@
+---
+layout: global
+title: Running with Cloudera and HortonWorks Distributions
+---
+
+Spark is fully compatible with all versions of Cloudera's Distribution Including Hadoop (CDH) and
+the Hortonworks Data Platform (HDP). There are a few things to keep in mind when running against
+these distributions:
+
+# Compile-time Hadoop Version
+When compiling Spark, you'll need to 
+[set the HADOOP_VERSION flag](http://localhost:4000/index.html#a-note-about-hadoop-versions). 
+The table below lists the corresponding HADOOP_VERSION for each CDH/HDP release. _Note_ that
+some Hadoop releases are binary compatible across client versions. This means the pre-built Spark
+distribution may "just work" without you needing to compile. That said, we recommend compiling with 
+the _exact_ Hadoop version you are running to avoid any compatibility errors.
+
+<table>
+  <tr valign="top">
+    <td>
+      <h3>CDH Releases</h3>
+      <table class="table" style="width:350px;">
+        <tr><th>Version</th><th>HADOOP_VERSION</th></tr>
+        <tr><td>CDH 4.X.X (YARN mode)</td><td>2.0.0-chd4.X.X</td></tr>
+        <tr><td>CDH 4.X.X</td><td>2.0.0-mr1-chd4.X.X</td></tr>
+        <tr><td>CDH 3u6</td><td>0.20.2-cdh3u6</td></tr>
+        <tr><td>CDH 3u5</td><td>0.20.2-cdh3u5</td></tr>
+        <tr><td>CDH 3u4</td><td>0.20.2-cdh3u4</td></tr>
+      </table>
+    </td>
+    <td>
+      <h3>HDP Releases</h3>
+      <table class="table" style="width:350px;">
+        <tr><th>Version</th><th>HADOOP_VERSION</th></tr>
+        <tr><td>HDP 1.3</td><td>1.2.0</td></tr>
+        <tr><td>HDP 1.2</td><td>1.1.2</td></tr>
+        <tr><td>HDP 1.1</td><td>1.0.3</td></tr>
+        <tr><td>HDP 1.0</td><td>1.0.3</td></tr>
+      </table>
+    </td>
+  </tr>
+</table>
+
+# Where to Run Spark
+As described in the [Hardware Provisioning](hardware-provisioning.html#storage-systems) guide,
+Spark can run in a variety of deployment modes:
+
+* Using dedicated set of Spark nodes in your cluster. These nodes should be co-located with your
+  Hadoop installation.
+* Running on the same nodes as an existing Hadoop installation, with a fixed amount memory and 
+  cores dedicated to Spark on each node.
+* Run Spark alongside Hadoop using a cluster resource manager, such as YARN or Mesos.
+
+These options are identical for those using CDH and HDP. Note that if you are running a YARN 
+cluster, you may still choose to run Spark on dedicated nodes. In this case, you should use 
+the `mr1` versions of HADOOP_HOME when compiling, not the YARN versions.
+
+# Inheriting Cluster Configuration
+If you plan to read and write from HDFS using Spark, it is good to include copies of two relevant 
+Hadoop configuration files in your $SPARK_HOME/conf directory. These are `hdfs-site.xml`, which 
+provides default behaviors for the HDFS client, and `core-site.xml`, which sets the default 
+filesystem name. The location of these configuration files varies across CDH and HDP versions, but
+a common location is inside of `/etc/hadoop/conf`. Some tools, such as Cloudera Manager, create
+configurations on-the-fly, but offer a mechanisms to download copies of them.
+
+If you can locate these files, copy them into $SPARK_HOME/conf/.
-- 
cgit v1.2.3


From 61c4762d454c7b62bdccd33a5aab5c1f7e8809c5 Mon Sep 17 00:00:00 2001
From: Patrick Wendell <pwendell@gmail.com>
Date: Sat, 7 Sep 2013 11:51:40 -0700
Subject: Changes based on feedback

---
 docs/cdh-hdp.md | 36 ++++++++++++++++++++++++------------
 1 file changed, 24 insertions(+), 12 deletions(-)

diff --git a/docs/cdh-hdp.md b/docs/cdh-hdp.md
index a4407a135d..679fb1100d 100644
--- a/docs/cdh-hdp.md
+++ b/docs/cdh-hdp.md
@@ -3,14 +3,17 @@ layout: global
 title: Running with Cloudera and HortonWorks Distributions
 ---
 
-Spark is fully compatible with all versions of Cloudera's Distribution Including Hadoop (CDH) and
-the Hortonworks Data Platform (HDP). There are a few things to keep in mind when running against
+Spark can run against all versions of Cloudera's Distribution Including Hadoop (CDH) and
+the Hortonworks Data Platform (HDP). There are a few things to keep in mind when using Spark with
 these distributions:
 
 # Compile-time Hadoop Version
 When compiling Spark, you'll need to 
-[set the HADOOP_VERSION flag](http://localhost:4000/index.html#a-note-about-hadoop-versions). 
-The table below lists the corresponding HADOOP_VERSION for each CDH/HDP release. _Note_ that
+[set the HADOOP_VERSION flag](http://localhost:4000/index.html#a-note-about-hadoop-versions):
+
+    HADOOP_VERSION=1.0.4 sbt/sbt assembly
+
+The table below lists the corresponding HADOOP_VERSION for each CDH/HDP release. Note that
 some Hadoop releases are binary compatible across client versions. This means the pre-built Spark
 distribution may "just work" without you needing to compile. That said, we recommend compiling with 
 the _exact_ Hadoop version you are running to avoid any compatibility errors.
@@ -51,16 +54,25 @@ Spark can run in a variety of deployment modes:
   cores dedicated to Spark on each node.
 * Run Spark alongside Hadoop using a cluster resource manager, such as YARN or Mesos.
 
-These options are identical for those using CDH and HDP. Note that if you are running a YARN 
-cluster, you may still choose to run Spark on dedicated nodes. In this case, you should use 
-the `mr1` versions of HADOOP_HOME when compiling, not the YARN versions.
+These options are identical for those using CDH and HDP. Note that if you have a YARN cluster,
+but still prefer to run Spark on a dedicated set of nodes rather than scheduling through YARN, 
+use `mr1` versions of HADOOP_HOME when compiling.
 
 # Inheriting Cluster Configuration
-If you plan to read and write from HDFS using Spark, it is good to include copies of two relevant 
-Hadoop configuration files in your $SPARK_HOME/conf directory. These are `hdfs-site.xml`, which 
-provides default behaviors for the HDFS client, and `core-site.xml`, which sets the default 
-filesystem name. The location of these configuration files varies across CDH and HDP versions, but
+If you plan to read and write from HDFS using Spark, there are two Hadoop configuration files that
+should be included on Spark's classpath:
+
+* `hdfs-site.xml`, which provides default behaviors for the HDFS client.
+* `core-site.xml`, which sets the default filesystem name.
+
+The location of these configuration files varies across CDH and HDP versions, but
 a common location is inside of `/etc/hadoop/conf`. Some tools, such as Cloudera Manager, create
 configurations on-the-fly, but offer a mechanisms to download copies of them.
 
-If you can locate these files, copy them into $SPARK_HOME/conf/.
+There are a few ways to make these files visible to Spark:
+
+* You can copy these files into `$SPARK_HOME/conf` and they will be included in Spark's
+classpath automatically.
+* If you are running Spark on the same nodes as Hadoop _and_ your distribution includes both
+`hdfs-site.xml` and `core-site.xml` in the same directory, you can set `HADOOP_CONF_DIR` 
+in `$SPARK_HOME/spark-env.sh` to that directory.
-- 
cgit v1.2.3


From 22b982d2bc76197c85ed50558a0fc676dffcc5ef Mon Sep 17 00:00:00 2001
From: Patrick Wendell <pwendell@gmail.com>
Date: Sat, 7 Sep 2013 14:37:54 -0700
Subject: File rename

---
 docs/_layouts/global.html                |  2 +-
 docs/cdh-hdp.md                          | 78 --------------------------------
 docs/hadoop-third-party-distributions.md | 76 +++++++++++++++++++++++++++++++
 3 files changed, 77 insertions(+), 79 deletions(-)
 delete mode 100644 docs/cdh-hdp.md
 create mode 100644 docs/hadoop-third-party-distributions.md

diff --git a/docs/_layouts/global.html b/docs/_layouts/global.html
index 3a3b8dce37..2f6bdcabe8 100755
--- a/docs/_layouts/global.html
+++ b/docs/_layouts/global.html
@@ -98,7 +98,7 @@
                             <ul class="dropdown-menu">
                                 <li><a href="configuration.html">Configuration</a></li>
                                 <li><a href="tuning.html">Tuning Guide</a></li>
-                                <li><a href="cdh-hdp.html">Running with CDH/HDP</a></li>
+                                <li><a href="hadoop-third-party-distributions.html">Running with CDH/HDP</a></li>
                                 <li><a href="hardware-provisioning.html">Hardware Provisioning</a></li>
                                 <li><a href="building-with-maven.html">Building Spark with Maven</a></li>
                                 <li><a href="contributing-to-spark.html">Contributing to Spark</a></li>
diff --git a/docs/cdh-hdp.md b/docs/cdh-hdp.md
deleted file mode 100644
index 679fb1100d..0000000000
--- a/docs/cdh-hdp.md
+++ /dev/null
@@ -1,78 +0,0 @@
----
-layout: global
-title: Running with Cloudera and HortonWorks Distributions
----
-
-Spark can run against all versions of Cloudera's Distribution Including Hadoop (CDH) and
-the Hortonworks Data Platform (HDP). There are a few things to keep in mind when using Spark with
-these distributions:
-
-# Compile-time Hadoop Version
-When compiling Spark, you'll need to 
-[set the HADOOP_VERSION flag](http://localhost:4000/index.html#a-note-about-hadoop-versions):
-
-    HADOOP_VERSION=1.0.4 sbt/sbt assembly
-
-The table below lists the corresponding HADOOP_VERSION for each CDH/HDP release. Note that
-some Hadoop releases are binary compatible across client versions. This means the pre-built Spark
-distribution may "just work" without you needing to compile. That said, we recommend compiling with 
-the _exact_ Hadoop version you are running to avoid any compatibility errors.
-
-<table>
-  <tr valign="top">
-    <td>
-      <h3>CDH Releases</h3>
-      <table class="table" style="width:350px;">
-        <tr><th>Version</th><th>HADOOP_VERSION</th></tr>
-        <tr><td>CDH 4.X.X (YARN mode)</td><td>2.0.0-chd4.X.X</td></tr>
-        <tr><td>CDH 4.X.X</td><td>2.0.0-mr1-chd4.X.X</td></tr>
-        <tr><td>CDH 3u6</td><td>0.20.2-cdh3u6</td></tr>
-        <tr><td>CDH 3u5</td><td>0.20.2-cdh3u5</td></tr>
-        <tr><td>CDH 3u4</td><td>0.20.2-cdh3u4</td></tr>
-      </table>
-    </td>
-    <td>
-      <h3>HDP Releases</h3>
-      <table class="table" style="width:350px;">
-        <tr><th>Version</th><th>HADOOP_VERSION</th></tr>
-        <tr><td>HDP 1.3</td><td>1.2.0</td></tr>
-        <tr><td>HDP 1.2</td><td>1.1.2</td></tr>
-        <tr><td>HDP 1.1</td><td>1.0.3</td></tr>
-        <tr><td>HDP 1.0</td><td>1.0.3</td></tr>
-      </table>
-    </td>
-  </tr>
-</table>
-
-# Where to Run Spark
-As described in the [Hardware Provisioning](hardware-provisioning.html#storage-systems) guide,
-Spark can run in a variety of deployment modes:
-
-* Using dedicated set of Spark nodes in your cluster. These nodes should be co-located with your
-  Hadoop installation.
-* Running on the same nodes as an existing Hadoop installation, with a fixed amount memory and 
-  cores dedicated to Spark on each node.
-* Run Spark alongside Hadoop using a cluster resource manager, such as YARN or Mesos.
-
-These options are identical for those using CDH and HDP. Note that if you have a YARN cluster,
-but still prefer to run Spark on a dedicated set of nodes rather than scheduling through YARN, 
-use `mr1` versions of HADOOP_HOME when compiling.
-
-# Inheriting Cluster Configuration
-If you plan to read and write from HDFS using Spark, there are two Hadoop configuration files that
-should be included on Spark's classpath:
-
-* `hdfs-site.xml`, which provides default behaviors for the HDFS client.
-* `core-site.xml`, which sets the default filesystem name.
-
-The location of these configuration files varies across CDH and HDP versions, but
-a common location is inside of `/etc/hadoop/conf`. Some tools, such as Cloudera Manager, create
-configurations on-the-fly, but offer a mechanisms to download copies of them.
-
-There are a few ways to make these files visible to Spark:
-
-* You can copy these files into `$SPARK_HOME/conf` and they will be included in Spark's
-classpath automatically.
-* If you are running Spark on the same nodes as Hadoop _and_ your distribution includes both
-`hdfs-site.xml` and `core-site.xml` in the same directory, you can set `HADOOP_CONF_DIR` 
-in `$SPARK_HOME/spark-env.sh` to that directory.
diff --git a/docs/hadoop-third-party-distributions.md b/docs/hadoop-third-party-distributions.md
new file mode 100644
index 0000000000..9f4f354525
--- /dev/null
+++ b/docs/hadoop-third-party-distributions.md
@@ -0,0 +1,76 @@
+---
+layout: global
+title: Running with Cloudera and HortonWorks Distributions
+---
+
+Spark can run against all versions of Cloudera's Distribution Including Hadoop (CDH) and
+the Hortonworks Data Platform (HDP). There are a few things to keep in mind when using Spark with
+these distributions:
+
+# Compile-time Hadoop Version
+When compiling Spark, you'll need to 
+[set the HADOOP_VERSION flag](http://localhost:4000/index.html#a-note-about-hadoop-versions):
+
+    HADOOP_VERSION=1.0.4 sbt/sbt assembly
+
+The table below lists the corresponding HADOOP_VERSION for each CDH/HDP release. Note that
+some Hadoop releases are binary compatible across client versions. This means the pre-built Spark
+distribution may "just work" without you needing to compile. That said, we recommend compiling with 
+the _exact_ Hadoop version you are running to avoid any compatibility errors.
+
+<table>
+  <tr valign="top">
+    <td>
+      <h3>CDH Releases</h3>
+      <table class="table" style="width:350px;">
+        <tr><th>Version</th><th>HADOOP_VERSION</th></tr>
+        <tr><td>CDH 4.X.X (YARN mode)</td><td>2.0.0-chd4.X.X</td></tr>
+        <tr><td>CDH 4.X.X</td><td>2.0.0-mr1-chd4.X.X</td></tr>
+        <tr><td>CDH 3u6</td><td>0.20.2-cdh3u6</td></tr>
+        <tr><td>CDH 3u5</td><td>0.20.2-cdh3u5</td></tr>
+        <tr><td>CDH 3u4</td><td>0.20.2-cdh3u4</td></tr>
+      </table>
+    </td>
+    <td>
+      <h3>HDP Releases</h3>
+      <table class="table" style="width:350px;">
+        <tr><th>Version</th><th>HADOOP_VERSION</th></tr>
+        <tr><td>HDP 1.3</td><td>1.2.0</td></tr>
+        <tr><td>HDP 1.2</td><td>1.1.2</td></tr>
+        <tr><td>HDP 1.1</td><td>1.0.3</td></tr>
+        <tr><td>HDP 1.0</td><td>1.0.3</td></tr>
+      </table>
+    </td>
+  </tr>
+</table>
+
+# Where to Run Spark
+As described in the [Hardware Provisioning](hardware-provisioning.html#storage-systems) guide,
+Spark can run in a variety of deployment modes:
+
+* Using dedicated set of Spark nodes in your cluster. These nodes should be co-located with your
+  Hadoop installation.
+* Running on the same nodes as an existing Hadoop installation, with a fixed amount memory and 
+  cores dedicated to Spark on each node.
+* Run Spark alongside Hadoop using a cluster resource manager, such as YARN or Mesos.
+
+These options are identical for those using CDH and HDP. 
+
+# Inheriting Cluster Configuration
+If you plan to read and write from HDFS using Spark, there are two Hadoop configuration files that
+should be included on Spark's classpath:
+
+* `hdfs-site.xml`, which provides default behaviors for the HDFS client.
+* `core-site.xml`, which sets the default filesystem name.
+
+The location of these configuration files varies across CDH and HDP versions, but
+a common location is inside of `/etc/hadoop/conf`. Some tools, such as Cloudera Manager, create
+configurations on-the-fly, but offer a mechanisms to download copies of them.
+
+There are a few ways to make these files visible to Spark:
+
+* You can copy these files into `$SPARK_HOME/conf` and they will be included in Spark's
+classpath automatically.
+* If you are running Spark on the same nodes as Hadoop _and_ your distribution includes both
+`hdfs-site.xml` and `core-site.xml` in the same directory, you can set `HADOOP_CONF_DIR` 
+in `$SPARK_HOME/spark-env.sh` to that directory.
-- 
cgit v1.2.3