diff options
| author | Brian O'Neill <bone@alumni.brown.edu> | 2016-05-12 20:10:33 +0100 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-05-12 20:10:33 +0100 |
| commit | 81e3bfc16c6cfbf2f9f2c6c32ed651b8450795ba (patch) | |
| tree | a6a8e3f33ec82702d26c5eeeeaa6addf7caca46f /external/kinesis-asl-assembly | |
| parent | 81bf870848cf9faeec5ab2d40acff27085466698 (diff) | |
| download | spark-81e3bfc16c6cfbf2f9f2c6c32ed651b8450795ba.tar.gz spark-81e3bfc16c6cfbf2f9f2c6c32ed651b8450795ba.tar.bz2 spark-81e3bfc16c6cfbf2f9f2c6c32ed651b8450795ba.zip | |
[SPARK-14421] Upgrades protobuf dependency to 2.6.1 for the new version of KCL, and…
## What changes were proposed in this pull request?
When running with Kinesis Consumer Library (KCL), against a stream that contains aggregated data, the KCL needs access to protobuf to de-aggregate the records. Without this patch, that results in the following error message:
```
Caused by: java.lang.ClassNotFoundException: com.google.protobuf.ProtocolStringList
```
This PR upgrades the protobuf dependency within the kinesis-asl-assembly, and relocates that package (as not to conflict with Spark's use of 2.5.0), which fixes the above CNFE.
## How was this patch tested?
Used kinesis word count example against a stream containing aggregated data.
See: SPARK-14421
Author: Brian O'Neill <bone@alumni.brown.edu>
Closes #13054 from boneill42/protobuf-relocation-for-kcl.
Diffstat (limited to 'external/kinesis-asl-assembly')
| -rw-r--r-- | external/kinesis-asl-assembly/pom.xml | 16 |
1 files changed, 15 insertions, 1 deletions
diff --git a/external/kinesis-asl-assembly/pom.xml b/external/kinesis-asl-assembly/pom.xml index e057b78abd..6fb88ebae5 100644 --- a/external/kinesis-asl-assembly/pom.xml +++ b/external/kinesis-asl-assembly/pom.xml @@ -63,7 +63,12 @@ <dependency> <groupId>com.google.protobuf</groupId> <artifactId>protobuf-java</artifactId> - <scope>provided</scope> + <version>2.6.1</version> + <!-- + We are being explicit about version here and overriding the + spark default of 2.5.0 because KCL appears to have introduced + a dependency on protobuf 2.6.1 somewhere between KCL 1.4.0 and 1.6.1. + --> </dependency> <dependency> <groupId>org.glassfish.jersey.core</groupId> @@ -147,6 +152,15 @@ <include>*:*</include> </includes> </artifactSet> + <relocations> + <relocation> + <pattern>com.google.protobuf</pattern> + <shadedPattern>kinesis.protobuf</shadedPattern> + <includes> + <include>com.google.protobuf.**</include> + </includes> + </relocation> + </relocations> <filters> <filter> <artifact>*:*</artifact> |