diff options
author | Jey Kottalam <jey@cs.berkeley.edu> | 2013-08-21 14:51:56 -0700 |
---|---|---|
committer | Jey Kottalam <jey@cs.berkeley.edu> | 2013-08-21 14:51:56 -0700 |
commit | 6585f49841ada637b0811e0aadcf93132fff7001 (patch) | |
tree | 1ad14c8bb54de48a96bfe6882436e9eb3d4175fb /docs/running-on-yarn.md | |
parent | 66e7a38a3229eeb6d980193048ebebcda1522acb (diff) | |
download | spark-6585f49841ada637b0811e0aadcf93132fff7001.tar.gz spark-6585f49841ada637b0811e0aadcf93132fff7001.tar.bz2 spark-6585f49841ada637b0811e0aadcf93132fff7001.zip |
Update build docs
Diffstat (limited to 'docs/running-on-yarn.md')
-rw-r--r-- | docs/running-on-yarn.md | 20 |
1 files changed, 7 insertions, 13 deletions
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 9c2cedfd88..6bada9bdd7 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -6,7 +6,7 @@ title: Launching Spark on YARN Experimental support for running over a [YARN (Hadoop NextGen)](http://hadoop.apache.org/docs/r2.0.2-alpha/hadoop-yarn/hadoop-yarn-site/YARN.html) cluster was added to Spark in version 0.6.0. This was merged into master as part of 0.7 effort. -To build spark core with YARN support, please use the hadoop2-yarn profile. +To build spark with YARN support, please use the hadoop2-yarn profile. Ex: mvn -Phadoop2-yarn clean install # Building spark core consolidated jar. @@ -15,18 +15,12 @@ We need a consolidated spark core jar (which bundles all the required dependenci This can be built either through sbt or via maven. - Building spark assembled jar via sbt. - It is a manual process of enabling it in project/SparkBuild.scala. -Please comment out the - HADOOP_VERSION, HADOOP_MAJOR_VERSION and HADOOP_YARN -variables before the line 'For Hadoop 2 YARN support' -Next, uncomment the subsequent 3 variable declaration lines (for these three variables) which enable hadoop yarn support. +Enable YARN support by setting `SPARK_WITH_YARN=true` when invoking sbt: -Assembly of the jar Ex: - - ./sbt/sbt clean assembly + SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_WITH_YARN=true ./sbt/sbt clean assembly The assembled jar would typically be something like : -`./core/target/spark-core-assembly-0.8.0-SNAPSHOT.jar` +`./yarn/target/spark-yarn-assembly-0.8.0-SNAPSHOT.jar` - Building spark assembled jar via Maven. @@ -34,16 +28,16 @@ The assembled jar would typically be something like : Something like this. Ex: - mvn -Phadoop2-yarn clean package -DskipTests=true + mvn -Phadoop2-yarn -Dhadoop.version=2.0.5-alpha clean package -DskipTests=true This will build the shaded (consolidated) jar. Typically something like : -`./repl-bin/target/spark-repl-bin-<VERSION>-shaded-hadoop2-yarn.jar` +`./yarn/target/spark-yarn-bin-<VERSION>-shaded.jar` # Preparations -- Building spark core assembled jar (see above). +- Building spark-yarn assembly (see above). - Your application code must be packaged into a separate JAR file. If you want to test out the YARN deployment mode, you can use the current Spark examples. A `spark-examples_{{site.SCALA_VERSION}}-{{site.SPARK_VERSION}}` file can be generated by running `sbt/sbt package`. NOTE: since the documentation you're reading is for Spark version {{site.SPARK_VERSION}}, we are assuming here that you have downloaded Spark {{site.SPARK_VERSION}} or checked it out of source control. If you are using a different version of Spark, the version numbers in the jar generated by the sbt package command will obviously be different. |