diff options
| author | Tathagata Das <tathagata.das1565@gmail.com> | 2014-08-22 21:34:48 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-08-22 21:34:48 -0700 |
| commit | 3004074152b7261c2a968bb8e94ec7c41a7b43c1 (patch) | |
| tree | 576634f49600123f38d2c8d398d7f07408563c35 /external/flume-sink/pom.xml | |
| parent | a5219db1ece20a1c45f927d4d0e085474f0bf34e (diff) | |
| download | spark-3004074152b7261c2a968bb8e94ec7c41a7b43c1.tar.gz spark-3004074152b7261c2a968bb8e94ec7c41a7b43c1.tar.bz2 spark-3004074152b7261c2a968bb8e94ec7c41a7b43c1.zip | |
[SPARK-3169] Removed dependency on spark streaming test from spark flume sink
Due to maven bug https://jira.codehaus.org/browse/MNG-1378, maven could not resolve spark streaming classes required by the spark-streaming test-jar dependency of external/flume-sink. There is no particular reason that the external/flume-sink has to depend on Spark Streaming at all, so I am eliminating this dependency. Also I have removed the exclusions present in the Flume dependencies, as there is no reason to exclude them (they were excluded in the external/flume module to prevent dependency collisions with Spark).
Since Jenkins will test the sbt build and the unit test, I only tested maven compilation locally.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #2101 from tdas/spark-sink-pom-fix and squashes the following commits:
8f42621 [Tathagata Das] Added Flume sink exclusions back, and added netty to test dependencies
93b559f [Tathagata Das] Removed dependency on spark streaming test from spark flume sink
Diffstat (limited to 'external/flume-sink/pom.xml')
| -rw-r--r-- | external/flume-sink/pom.xml | 18 |
1 files changed, 12 insertions, 6 deletions
diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index 0c68defa5e..19192e40a7 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -62,7 +62,7 @@ <groupId>org.apache.thrift</groupId> <artifactId>libthrift</artifactId> </exclusion> - </exclusions> + </exclusions> </dependency> <dependency> <groupId>org.scala-lang</groupId> @@ -71,13 +71,19 @@ <dependency> <groupId>org.scalatest</groupId> <artifactId>scalatest_${scala.binary.version}</artifactId> + <scope>test</scope> </dependency> <dependency> - <groupId>org.apache.spark</groupId> - <artifactId>spark-streaming_${scala.binary.version}</artifactId> - <version>${project.version}</version> - <type>test-jar</type> - <scope>test</scope> <!-- Need it only for tests, don't package it --> + <!-- + Netty explicitly added in test as it has been excluded from + Flume dependency (to avoid runtime problems when running with + Spark) but unit tests need it. Version of Netty on which + Flume 1.4.0 depends on is "3.4.0.Final" . + --> + <groupId>io.netty</groupId> + <artifactId>netty</artifactId> + <version>3.4.0.Final</version> + <scope>test</scope> </dependency> </dependencies> <build> |