diff options
author | Sean Owen <sowen@cloudera.com> | 2014-05-12 14:17:25 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-05-12 14:17:34 -0700 |
commit | e1852812ff9a8b0b9eb28990db0ad754ab80b08b (patch) | |
tree | 7586ebad349e29c363146881cb2ad1553095271f /yarn/stable | |
parent | 14515b416e9a00cb3a77c2645984265b30b05882 (diff) | |
download | spark-e1852812ff9a8b0b9eb28990db0ad754ab80b08b.tar.gz spark-e1852812ff9a8b0b9eb28990db0ad754ab80b08b.tar.bz2 spark-e1852812ff9a8b0b9eb28990db0ad754ab80b08b.zip |
SPARK-1802. Audit dependency graph when Spark is built with -Phive
This initial commit resolves the conflicts in the Hive profiles as noted in https://issues.apache.org/jira/browse/SPARK-1802 .
Most of the fix was to note that Hive drags in Avro, and so if the hive module depends on Spark's version of the `avro-*` dependencies, it will pull in our exclusions as needed too. But I found we need to copy some exclusions between the two Avro dependencies to get this right. And then had to squash some commons-logging intrusions.
This turned up another annoying find, that `hive-exec` is basically an "assembly" artifact that _also_ packages all of its transitive dependencies. This means the final assembly shows lots of collisions between itself and its dependencies, and even other project dependencies. I have a TODO to examine whether that is going to be a deal-breaker or not.
In the meantime I'm going to tack on a second commit to this PR that will also fix some similar, last collisions in the YARN profile.
Author: Sean Owen <sowen@cloudera.com>
Closes #744 from srowen/SPARK-1802 and squashes the following commits:
a856604 [Sean Owen] Resolve JAR version conflicts specific to Hive profile
(cherry picked from commit 8586bf564fe010dfc19ef26874472a6f85e355fb)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Diffstat (limited to 'yarn/stable')
0 files changed, 0 insertions, 0 deletions