diff options
| author | Reynold Xin <rxin@databricks.com> | 2015-12-21 22:15:52 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-12-21 22:15:52 -0800 |
| commit | 0a38637d05d2338503ecceacfb911a6da6d49538 (patch) | |
| tree | 080b3f2485cd1e2d93f23b764181415df795719d /docs/building-spark.md | |
| parent | 29cecd4a42f6969613e5b2a40f2724f99e7eec01 (diff) | |
| download | spark-0a38637d05d2338503ecceacfb911a6da6d49538.tar.gz spark-0a38637d05d2338503ecceacfb911a6da6d49538.tar.bz2 spark-0a38637d05d2338503ecceacfb911a6da6d49538.zip | |
[SPARK-11807] Remove support for Hadoop < 2.2
i.e. Hadoop 1 and Hadoop 2.0
Author: Reynold Xin <rxin@databricks.com>
Closes #10404 from rxin/SPARK-11807.
Diffstat (limited to 'docs/building-spark.md')
| -rw-r--r-- | docs/building-spark.md | 18 |
1 files changed, 4 insertions, 14 deletions
diff --git a/docs/building-spark.md b/docs/building-spark.md index 3d38edbdad..785988902d 100644 --- a/docs/building-spark.md +++ b/docs/building-spark.md @@ -33,13 +33,13 @@ to the `sharedSettings` val. See also [this PR](https://github.com/apache/spark/ # Building a Runnable Distribution -To create a Spark distribution like those distributed by the -[Spark Downloads](http://spark.apache.org/downloads.html) page, and that is laid out so as -to be runnable, use `make-distribution.sh` in the project root directory. It can be configured +To create a Spark distribution like those distributed by the +[Spark Downloads](http://spark.apache.org/downloads.html) page, and that is laid out so as +to be runnable, use `make-distribution.sh` in the project root directory. It can be configured with Maven profile settings and so on like the direct Maven build. Example: ./make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn - + For more information on usage, run `./make-distribution.sh --help` # Setting up Maven's Memory Usage @@ -74,7 +74,6 @@ Because HDFS is not protocol-compatible across versions, if you want to read fro <tr><th>Hadoop version</th><th>Profile required</th></tr> </thead> <tbody> - <tr><td>1.x to 2.1.x</td><td>hadoop-1</td></tr> <tr><td>2.2.x</td><td>hadoop-2.2</td></tr> <tr><td>2.3.x</td><td>hadoop-2.3</td></tr> <tr><td>2.4.x</td><td>hadoop-2.4</td></tr> @@ -82,15 +81,6 @@ Because HDFS is not protocol-compatible across versions, if you want to read fro </tbody> </table> -For Apache Hadoop versions 1.x, Cloudera CDH "mr1" distributions, and other Hadoop versions without YARN, use: - -{% highlight bash %} -# Apache Hadoop 1.2.1 -mvn -Dhadoop.version=1.2.1 -Phadoop-1 -DskipTests clean package - -# Cloudera CDH 4.2.0 with MapReduce v1 -mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -Phadoop-1 -DskipTests clean package -{% endhighlight %} You can enable the `yarn` profile and optionally set the `yarn.version` property if it is different from `hadoop.version`. Spark only supports YARN versions 2.2.0 and later. |