diff options
| author | Burak Yavuz <brkyvz@gmail.com> | 2015-11-09 17:18:49 -0800 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-11-09 17:18:49 -0800 |
| commit | 26062d22607e1f9854bc2588ba22a4e0f8bba48c (patch) | |
| tree | c9a3c0e2a0bf59a481af2637e700500799569de0 /pom.xml | |
| parent | 61f9c8711c79f35d67b0456155866da316b131d9 (diff) | |
| download | spark-26062d22607e1f9854bc2588ba22a4e0f8bba48c.tar.gz spark-26062d22607e1f9854bc2588ba22a4e0f8bba48c.tar.bz2 spark-26062d22607e1f9854bc2588ba22a4e0f8bba48c.zip | |
[SPARK-11198][STREAMING][KINESIS] Support de-aggregation of records during recovery
While the KCL handles de-aggregation during the regular operation, during recovery we use the lower level api, and therefore need to de-aggregate the records.
tdas Testing is an issue, we need protobuf magic to do the aggregated records. Maybe we could depend on KPL for tests?
Author: Burak Yavuz <brkyvz@gmail.com>
Closes #9403 from brkyvz/kinesis-deaggregation.
Diffstat (limited to 'pom.xml')
| -rw-r--r-- | pom.xml | 2 |
1 files changed, 2 insertions, 0 deletions
@@ -154,6 +154,8 @@ <jets3t.version>0.7.1</jets3t.version> <aws.java.sdk.version>1.9.40</aws.java.sdk.version> <aws.kinesis.client.version>1.4.0</aws.kinesis.client.version> + <!-- the producer is used in tests --> + <aws.kinesis.producer.version>0.10.1</aws.kinesis.producer.version> <!-- org.apache.httpcomponents/httpclient--> <commons.httpclient.version>4.3.2</commons.httpclient.version> <!-- commons-httpclient/commons-httpclient--> |