diff options
| author | Sandy Ryza <sandy@cloudera.com> | 2014-03-11 22:39:17 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-03-11 22:39:17 -0700 |
| commit | 2409af9dcf238e1ad87080a389e05a696c41dc72 (patch) | |
| tree | 2c65a789a99b7e95389cf766140eecf55af99f31 /docs/building-with-maven.md | |
| parent | 16788a654246067fd966033b5dc9bc0d4c759b70 (diff) | |
| download | spark-2409af9dcf238e1ad87080a389e05a696c41dc72.tar.gz spark-2409af9dcf238e1ad87080a389e05a696c41dc72.tar.bz2 spark-2409af9dcf238e1ad87080a389e05a696c41dc72.zip | |
SPARK-1064
This reopens PR 649 from incubator-spark against the new repo
Author: Sandy Ryza <sandy@cloudera.com>
Closes #102 from sryza/sandy-spark-1064 and squashes the following commits:
270e490 [Sandy Ryza] Handle different application classpath variables in different versions
88b04e0 [Sandy Ryza] SPARK-1064. Make it possible to run on YARN without bundling Hadoop jars in Spark assembly
Diffstat (limited to 'docs/building-with-maven.md')
| -rw-r--r-- | docs/building-with-maven.md | 6 |
1 files changed, 6 insertions, 0 deletions
diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md index d3bc34e68b..730a6e7932 100644 --- a/docs/building-with-maven.md +++ b/docs/building-with-maven.md @@ -88,3 +88,9 @@ Running only java 8 tests and nothing else. Java 8 tests are run when -Pjava8-tests profile is enabled, they will run in spite of -DskipTests. For these tests to run your system must have a JDK 8 installation. If you have JDK 8 installed but it is not the system default, you can set JAVA_HOME to point to JDK 8 before running the tests. + +## Packaging without Hadoop dependencies for deployment on YARN ## + +The assembly jar produced by "mvn package" will, by default, include all of Spark's dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor classpaths: the version packaged in the Spark assembly and the version on each node, included with yarn.application.classpath. The "hadoop-provided" profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself. + + |