diff options
| author | Raymond Liu <raymond.liu@intel.com> | 2013-12-06 10:36:59 +0800 |
|---|---|---|
| committer | Raymond Liu <raymond.liu@intel.com> | 2014-01-03 12:12:38 +0800 |
| commit | 7815a3ace97848b1d4bab4ef6823fc6d55cfb72e (patch) | |
| tree | ae4cd098d8a5643677d8e42f8c9d0179caab391e | |
| parent | be343d2a567b8ba3824c5e186e046f96ff77e334 (diff) | |
| download | spark-7815a3ace97848b1d4bab4ef6823fc6d55cfb72e.tar.gz spark-7815a3ace97848b1d4bab4ef6823fc6d55cfb72e.tar.bz2 spark-7815a3ace97848b1d4bab4ef6823fc6d55cfb72e.zip | |
Update maven build documentation
| -rw-r--r-- | docs/building-with-maven.md | 10 | ||||
| -rw-r--r-- | docs/running-on-yarn.md | 2 |
2 files changed, 4 insertions, 8 deletions
diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md index c709001632..699b10b188 100644 --- a/docs/building-with-maven.md +++ b/docs/building-with-maven.md @@ -37,7 +37,7 @@ For Apache Hadoop versions 1.x, Cloudera CDH MRv1, and other Hadoop versions wit # Cloudera CDH 4.2.0 with MapReduce v1 $ mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests clean package -For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN, you should enable the "hadoop2-yarn" profile and set the "yarn.version" property: +For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN, you should enable the "hadoop2-yarn" or "hadoop2.2-yarn" profile and set the "hadoop.version", "yarn.version" property: # Apache Hadoop 2.0.5-alpha $ mvn -Phadoop2-yarn -Dhadoop.version=2.0.5-alpha -Dyarn.version=2.0.5-alpha -DskipTests clean package @@ -45,12 +45,8 @@ For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with # Cloudera CDH 4.2.0 with MapReduce v2 $ mvn -Phadoop2-yarn -Dhadoop.version=2.0.0-cdh4.2.0 -Dyarn.version=2.0.0-chd4.2.0 -DskipTests clean package -Hadoop versions 2.2.x and newer can be built by setting the ```new-yarn``` and the ```yarn.version``` as follows: - - # Apache Hadoop 2.2.X and newer - $ mvn -Dyarn.version=2.2.0 -Dhadoop.version=2.2.0 -Pnew-yarn - -The build process handles Hadoop 2.2.x as a special case that uses the directory ```new-yarn```, which supports the new YARN API. Furthermore, for this version, the build depends on artifacts published by the spark-project to enable Akka 2.0.5 to work with protobuf 2.5. + # Apache Hadoop 2.2.X ( e.g. 2.2.0 as below ) and newer + $ mvn -Phadoop2.2-yarn -Dhadoop.version=2.2.0 -Dyarn.version=2.2.0 -DskipTests clean package ## Spark Tests in Maven ## diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 5f995ce51b..f47098554e 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -116,7 +116,7 @@ For example: # Building Spark for Hadoop/YARN 2.2.x -Hadoop 2.2.x users must build Spark and publish it locally. The SBT build process handles Hadoop 2.2.x as a special case. This version of Hadoop has new YARN API changes and depends on a Protobuf version (2.5) that is not compatible with the Akka version (2.0.5) that Spark uses. Therefore, if the Hadoop version (e.g. set through ```SPARK_HADOOP_VERSION```) starts with 2.2.0 or higher then the build process will depend on Akka artifacts distributed by the Spark project compatible with Protobuf 2.5. +Hadoop 2.2.x users must build Spark and publish it locally. The SBT build process handles Hadoop 2.2.x as a special case. This version of Hadoop has new YARN API changes and depends on a Protobuf version (2.5). See [Building Spark with Maven](building-with-maven.html) for instructions on how to build Spark using the Maven process. |