diff options
| author | BrianLondon <brian@seatgeek.com> | 2016-01-05 23:15:07 +0000 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-01-05 23:15:07 +0000 |
| commit | ff89975543b153d0d235c0cac615d45b34aa8fe7 (patch) | |
| tree | 0287ec4ff1b9d4143e80f02798667a8ceea28b6f /extras | |
| parent | 78015a8b7cc316343e302eeed6fe30af9f2961e8 (diff) | |
| download | spark-ff89975543b153d0d235c0cac615d45b34aa8fe7.tar.gz spark-ff89975543b153d0d235c0cac615d45b34aa8fe7.tar.bz2 spark-ff89975543b153d0d235c0cac615d45b34aa8fe7.zip | |
[SPARK-12453][STREAMING] Remove explicit dependency on aws-java-sdk
Successfully ran kinesis demo on a live, aws hosted kinesis stream against master and 1.6 branches. For reasons I don't entirely understand it required a manual merge to 1.5 which I did as shown here: https://github.com/BrianLondon/spark/commit/075c22e89bc99d5e99be21f40e0d72154a1e23a2
The demo ran successfully on the 1.5 branch as well.
According to `mvn dependency:tree` it is still pulling a fairly old version of the aws-java-sdk (1.9.37), but this appears to have fixed the kinesis regression in 1.5.2.
Author: BrianLondon <brian@seatgeek.com>
Closes #10492 from BrianLondon/remove-only.
Diffstat (limited to 'extras')
| -rw-r--r-- | extras/kinesis-asl/pom.xml | 5 | ||||
| -rw-r--r-- | extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala | 1 |
2 files changed, 1 insertions, 5 deletions
diff --git a/extras/kinesis-asl/pom.xml b/extras/kinesis-asl/pom.xml index 3c5722502e..20e2c5e0ff 100644 --- a/extras/kinesis-asl/pom.xml +++ b/extras/kinesis-asl/pom.xml @@ -61,11 +61,6 @@ </dependency> <dependency> <groupId>com.amazonaws</groupId> - <artifactId>aws-java-sdk</artifactId> - <version>${aws.java.sdk.version}</version> - </dependency> - <dependency> - <groupId>com.amazonaws</groupId> <artifactId>amazon-kinesis-producer</artifactId> <version>${aws.kinesis.producer.version}</version> <scope>test</scope> diff --git a/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala b/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala index 80edda59e1..abb9b6cd32 100644 --- a/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala +++ b/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala @@ -185,6 +185,7 @@ private[kinesis] class KinesisReceiver[T]( workerThread.setName(s"Kinesis Receiver ${streamId}") workerThread.setDaemon(true) workerThread.start() + logInfo(s"Started receiver with workerId $workerId") } |