diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2016-01-10 19:59:01 -0800 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2016-01-10 19:59:01 -0800 |
| commit | 3ab0138b0fe0f9208b4b476855294a7c729583b7 (patch) | |
| tree | c2946df3470e825087fa61f4ca0928dd45736942 /dev/test-dependencies.sh | |
| parent | 3119206b7188c23055621dfeaf6874f21c711a82 (diff) | |
| download | spark-3ab0138b0fe0f9208b4b476855294a7c729583b7.tar.gz spark-3ab0138b0fe0f9208b4b476855294a7c729583b7.tar.bz2 spark-3ab0138b0fe0f9208b4b476855294a7c729583b7.zip | |
[SPARK-12734][BUILD] Fix Netty exclusion and use Maven Enforcer to prevent future bugs
Netty classes are published under multiple artifacts with different names, so our build needs to exclude the `io.netty:netty` and `org.jboss.netty:netty` versions of the Netty artifact. However, our existing exclusions were incomplete, leading to situations where duplicate Netty classes would wind up on the classpath and cause compile errors (or worse).
This patch fixes the exclusion issue by adding more exclusions and uses Maven Enforcer's [banned dependencies](https://maven.apache.org/enforcer/enforcer-rules/bannedDependencies.html) rule to prevent these classes from accidentally being reintroduced. I also updated `dev/test-dependencies.sh` to run `mvn validate` so that the enforcer rules can run as part of pull request builds.
/cc rxin srowen pwendell. I'd like to backport at least the exclusion portion of this fix to `branch-1.5` in order to fix the documentation publishing job, which fails nondeterministically due to incompatible versions of Netty classes taking precedence on the compile-time classpath.
Author: Josh Rosen <rosenville@gmail.com>
Author: Josh Rosen <joshrosen@databricks.com>
Closes #10672 from JoshRosen/enforce-netty-exclusions.
Diffstat (limited to 'dev/test-dependencies.sh')
| -rwxr-xr-x | dev/test-dependencies.sh | 17 |
1 files changed, 4 insertions, 13 deletions
diff --git a/dev/test-dependencies.sh b/dev/test-dependencies.sh index 424ce6ad76..def87aa408 100755 --- a/dev/test-dependencies.sh +++ b/dev/test-dependencies.sh @@ -70,19 +70,10 @@ $MVN -q versions:set -DnewVersion=$TEMP_VERSION -DgenerateBackupPoms=false > /de # Generate manifests for each Hadoop profile: for HADOOP_PROFILE in "${HADOOP_PROFILES[@]}"; do echo "Performing Maven install for $HADOOP_PROFILE" - $MVN $HADOOP2_MODULE_PROFILES -P$HADOOP_PROFILE jar:jar install:install -q \ - -pl '!assembly' \ - -pl '!examples' \ - -pl '!external/flume-assembly' \ - -pl '!external/kafka-assembly' \ - -pl '!external/twitter' \ - -pl '!external/flume' \ - -pl '!external/mqtt' \ - -pl '!external/mqtt-assembly' \ - -pl '!external/zeromq' \ - -pl '!external/kafka' \ - -pl '!tags' \ - -DskipTests + $MVN $HADOOP2_MODULE_PROFILES -P$HADOOP_PROFILE jar:jar jar:test-jar install:install -q + + echo "Performing Maven validate for $HADOOP_PROFILE" + $MVN $HADOOP2_MODULE_PROFILES -P$HADOOP_PROFILE validate -q echo "Generating dependency manifest for $HADOOP_PROFILE" mkdir -p dev/pr-deps |