diff options
author | Josh Rosen <joshrosen@databricks.com> | 2015-11-10 10:14:19 -0800 |
---|---|---|
committer | Michael Armbrust <michael@databricks.com> | 2015-11-10 10:14:19 -0800 |
commit | 689386b1c60997e4505749915f7005a52c207de2 (patch) | |
tree | f3d1184de25b306519883fc8c00fb08948a3939c /CONTRIBUTING.md | |
parent | a81f47ff7498e7063c855ccf75bba81ab101b43e (diff) | |
download | spark-689386b1c60997e4505749915f7005a52c207de2.tar.gz spark-689386b1c60997e4505749915f7005a52c207de2.tar.bz2 spark-689386b1c60997e4505749915f7005a52c207de2.zip |
[SPARK-7841][BUILD] Stop using retrieveManaged to retrieve dependencies in SBT
This patch modifies Spark's SBT build so that it no longer uses `retrieveManaged` / `lib_managed` to store its dependencies. The motivations for this change are nicely described on the JIRA ticket ([SPARK-7841](https://issues.apache.org/jira/browse/SPARK-7841)); my personal interest in doing this stems from the fact that `lib_managed` has caused me some pain while debugging dependency issues in another PR of mine.
Removing our use of `lib_managed` would be trivial except for one snag: the Datanucleus JARs, required by Spark SQL's Hive integration, cannot be included in assembly JARs due to problems with merging OSGI `plugin.xml` files. As a result, several places in the packaging and deployment pipeline assume that these Datanucleus JARs are copied to `lib_managed/jars`. In the interest of maintaining compatibility, I have chosen to retain the `lib_managed/jars` directory _only_ for these Datanucleus JARs and have added custom code to `SparkBuild.scala` to automatically copy those JARs to that folder as part of the `assembly` task.
`dev/mima` also depended on `lib_managed` in a hacky way in order to set classpaths when generating MiMa excludes; I've updated this to obtain the classpaths directly from SBT instead.
/cc dragos marmbrus pwendell srowen
Author: Josh Rosen <joshrosen@databricks.com>
Closes #9575 from JoshRosen/SPARK-7841.
Diffstat (limited to 'CONTRIBUTING.md')
0 files changed, 0 insertions, 0 deletions