diff options
| author | Steve Loughran <stevel@hortonworks.com> | 2016-02-09 11:01:47 -0800 |
|---|---|---|
| committer | Marcelo Vanzin <vanzin@cloudera.com> | 2016-02-09 11:01:47 -0800 |
| commit | 34d0b70b309f16af263eb4e6d7c36e2ea170bc67 (patch) | |
| tree | b7a2c341e4f1561f97b9302feefc0fd046802832 /network | |
| parent | 68ed3632c56389ab3ff4ea5d73c575f224dab4f6 (diff) | |
| download | spark-34d0b70b309f16af263eb4e6d7c36e2ea170bc67.tar.gz spark-34d0b70b309f16af263eb4e6d7c36e2ea170bc67.tar.bz2 spark-34d0b70b309f16af263eb4e6d7c36e2ea170bc67.zip | |
[SPARK-12807][YARN] Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
Patch to
1. Shade jackson 2.x in spark-yarn-shuffle JAR: core, databind, annotation
2. Use maven antrun to verify the JAR has the renamed classes
Being Maven-based, I don't know if the verification phase kicks in on an SBT/jenkins build. It will on a `mvn install`
Author: Steve Loughran <stevel@hortonworks.com>
Closes #10780 from steveloughran/stevel/patches/SPARK-12807-master-shuffle.
Diffstat (limited to 'network')
| -rw-r--r-- | network/yarn/pom.xml | 49 |
1 files changed, 48 insertions, 1 deletions
diff --git a/network/yarn/pom.xml b/network/yarn/pom.xml index a28785b16e..3cb44324f2 100644 --- a/network/yarn/pom.xml +++ b/network/yarn/pom.xml @@ -35,6 +35,8 @@ <sbt.project.name>network-yarn</sbt.project.name> <!-- Make sure all Hadoop dependencies are provided to avoid repackaging. --> <hadoop.deps.scope>provided</hadoop.deps.scope> + <shuffle.jar>${project.build.directory}/scala-${scala.binary.version}/spark-${project.version}-yarn-shuffle.jar</shuffle.jar> + <shade>org/spark-project/</shade> </properties> <dependencies> @@ -70,7 +72,7 @@ <artifactId>maven-shade-plugin</artifactId> <configuration> <shadedArtifactAttached>false</shadedArtifactAttached> - <outputFile>${project.build.directory}/scala-${scala.binary.version}/spark-${project.version}-yarn-shuffle.jar</outputFile> + <outputFile>${shuffle.jar}</outputFile> <artifactSet> <includes> <include>*:*</include> @@ -86,6 +88,15 @@ </excludes> </filter> </filters> + <relocations> + <relocation> + <pattern>com.fasterxml.jackson</pattern> + <shadedPattern>org.spark-project.com.fasterxml.jackson</shadedPattern> + <includes> + <include>com.fasterxml.jackson.**</include> + </includes> + </relocation> + </relocations> </configuration> <executions> <execution> @@ -96,6 +107,42 @@ </execution> </executions> </plugin> + + <!-- probes to validate that those dependencies which must be shaded are --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-antrun-plugin</artifactId> + <executions> + <execution> + <phase>verify</phase> + <goals> + <goal>run</goal> + </goals> + <configuration> + <target> + <macrodef name="shaded"> + <attribute name="resource"/> + <sequential> + <fail message="Not found ${shade}@{resource}"> + <condition> + <not> + <resourceexists> + <zipentry zipfile="${shuffle.jar}" name="${shade}@{resource}"/> + </resourceexists> + </not> + </condition> + </fail> + </sequential> + </macrodef> + <echo>Verifying dependency shading</echo> + <shaded resource="com/fasterxml/jackson/core/JsonParser.class" /> + <shaded resource="com/fasterxml/jackson/annotation/JacksonAnnotation.class" /> + <shaded resource="com/fasterxml/jackson/databind/JsonSerializer.class" /> + </target> + </configuration> + </execution> + </executions> + </plugin> </plugins> </build> </project> |