diff options
| author | Patrick Wendell <pwendell@gmail.com> | 2014-03-02 15:18:19 -0800 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-03-02 15:18:19 -0800 |
| commit | c3f5e075335a65ea522b2f76716921ec056c52ed (patch) | |
| tree | f4a19d268063a07fbaa63564d67c56cc210beeb4 /yarn | |
| parent | fd31adbf27d824f00f62646e13c23f632d1b77d3 (diff) | |
| download | spark-c3f5e075335a65ea522b2f76716921ec056c52ed.tar.gz spark-c3f5e075335a65ea522b2f76716921ec056c52ed.tar.bz2 spark-c3f5e075335a65ea522b2f76716921ec056c52ed.zip | |
SPARK-1121: Include avro for yarn-alpha builds
This lets us explicitly include Avro based on a profile for 0.23.X
builds. It makes me sad how convoluted it is to express this logic
in Maven. @tgraves and @sryza curious if this works for you.
I'm also considering just reverting to how it was before. The only
real problem was that Spark advertised a dependency on Avro
even though it only really depends transitively on Avro through
other deps.
Author: Patrick Wendell <pwendell@gmail.com>
Closes #49 from pwendell/avro-build-fix and squashes the following commits:
8d6ee92 [Patrick Wendell] SPARK-1121: Add avro to yarn-alpha profile
Diffstat (limited to 'yarn')
| -rw-r--r-- | yarn/alpha/pom.xml | 14 | ||||
| -rw-r--r-- | yarn/pom.xml | 17 | ||||
| -rw-r--r-- | yarn/stable/pom.xml | 14 |
3 files changed, 37 insertions, 8 deletions
diff --git a/yarn/alpha/pom.xml b/yarn/alpha/pom.xml index e076ca1d44..bfe12ecec0 100644 --- a/yarn/alpha/pom.xml +++ b/yarn/alpha/pom.xml @@ -24,6 +24,20 @@ <relativePath>../pom.xml</relativePath> </parent> + <profiles> + <profile> + <!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around + a Hadoop 0.23.X issue --> + <id>yarn-alpha</id> + <dependencies> + <dependency> + <groupId>org.apache.avro</groupId> + <artifactId>avro</artifactId> + </dependency> + </dependencies> + </profile> + </profiles> + <groupId>org.apache.spark</groupId> <artifactId>spark-yarn-alpha_2.10</artifactId> <packaging>jar</packaging> diff --git a/yarn/pom.xml b/yarn/pom.xml index be51679292..35e31760c1 100644 --- a/yarn/pom.xml +++ b/yarn/pom.xml @@ -53,14 +53,6 @@ <version>${yarn.version}</version> </dependency> <dependency> - <groupId>org.apache.avro</groupId> - <artifactId>avro</artifactId> - </dependency> - <dependency> - <groupId>org.apache.avro</groupId> - <artifactId>avro-ipc</artifactId> - </dependency> - <dependency> <groupId>org.scalatest</groupId> <artifactId>scalatest_${scala.binary.version}</artifactId> <scope>test</scope> @@ -78,6 +70,15 @@ <modules> <module>alpha</module> </modules> + + <!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around + a Hadoop 0.23.X issue --> + <dependencies> + <dependency> + <groupId>org.apache.avro</groupId> + <artifactId>avro</artifactId> + </dependency> + </dependencies> </profile> <profile> diff --git a/yarn/stable/pom.xml b/yarn/stable/pom.xml index 0780f251b5..9d68603251 100644 --- a/yarn/stable/pom.xml +++ b/yarn/stable/pom.xml @@ -24,6 +24,20 @@ <relativePath>../pom.xml</relativePath> </parent> + <profiles> + <profile> + <!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around + a Hadoop 0.23.X issue --> + <id>yarn-alpha</id> + <dependencies> + <dependency> + <groupId>org.apache.avro</groupId> + <artifactId>avro</artifactId> + </dependency> + </dependencies> + </profile> + </profiles> + <groupId>org.apache.spark</groupId> <artifactId>spark-yarn_2.10</artifactId> <packaging>jar</packaging> |