diff options
| author | Sean Owen <sowen@cloudera.com> | 2014-05-12 14:17:25 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-05-12 14:17:25 -0700 |
| commit | 8586bf564fe010dfc19ef26874472a6f85e355fb (patch) | |
| tree | 793635f2e968bf59674056d3579c291fdaa9700d /sql/hive/pom.xml | |
| parent | 7120a2979d0a9f0f54a88b2416be7ca10e74f409 (diff) | |
| download | spark-8586bf564fe010dfc19ef26874472a6f85e355fb.tar.gz spark-8586bf564fe010dfc19ef26874472a6f85e355fb.tar.bz2 spark-8586bf564fe010dfc19ef26874472a6f85e355fb.zip | |
SPARK-1802. Audit dependency graph when Spark is built with -Phive
This initial commit resolves the conflicts in the Hive profiles as noted in https://issues.apache.org/jira/browse/SPARK-1802 .
Most of the fix was to note that Hive drags in Avro, and so if the hive module depends on Spark's version of the `avro-*` dependencies, it will pull in our exclusions as needed too. But I found we need to copy some exclusions between the two Avro dependencies to get this right. And then had to squash some commons-logging intrusions.
This turned up another annoying find, that `hive-exec` is basically an "assembly" artifact that _also_ packages all of its transitive dependencies. This means the final assembly shows lots of collisions between itself and its dependencies, and even other project dependencies. I have a TODO to examine whether that is going to be a deal-breaker or not.
In the meantime I'm going to tack on a second commit to this PR that will also fix some similar, last collisions in the YARN profile.
Author: Sean Owen <sowen@cloudera.com>
Closes #744 from srowen/SPARK-1802 and squashes the following commits:
a856604 [Sean Owen] Resolve JAR version conflicts specific to Hive profile
Diffstat (limited to 'sql/hive/pom.xml')
| -rw-r--r-- | sql/hive/pom.xml | 21 |
1 files changed, 21 insertions, 0 deletions
diff --git a/sql/hive/pom.xml b/sql/hive/pom.xml index 889d249146..8b32451d76 100644 --- a/sql/hive/pom.xml +++ b/sql/hive/pom.xml @@ -52,6 +52,12 @@ <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>${hive.version}</version> + <exclusions> + <exclusion> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + </exclusion> + </exclusions> </dependency> <dependency> <groupId>org.codehaus.jackson</groupId> @@ -61,6 +67,21 @@ <groupId>org.apache.hive</groupId> <artifactId>hive-serde</artifactId> <version>${hive.version}</version> + <exclusions> + <exclusion> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + </exclusion> + <exclusion> + <groupId>commons-logging</groupId> + <artifactId>commons-logging-api</artifactId> + </exclusion> + </exclusions> + </dependency> + <!-- hive-serde already depends on avro, but this brings in customized config of avro deps from parent --> + <dependency> + <groupId>org.apache.avro</groupId> + <artifactId>avro</artifactId> </dependency> <dependency> <groupId>org.scalatest</groupId> |