diff options
| author | Hari Shreedharan <hshreedharan@apache.org> | 2015-02-04 14:20:44 -0800 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-02-04 14:20:44 -0800 |
| commit | f0500f9fa378d81e4b4038a66a40eee15806b677 (patch) | |
| tree | d21e8c989705e05acf4c59c6c920af18c6e38010 /docs/java-programming-guide.md | |
| parent | b0c0021953826bccaee818a54afc44e8bdfa8572 (diff) | |
| download | spark-f0500f9fa378d81e4b4038a66a40eee15806b677.tar.gz spark-f0500f9fa378d81e4b4038a66a40eee15806b677.tar.bz2 spark-f0500f9fa378d81e4b4038a66a40eee15806b677.zip | |
[SPARK-4707][STREAMING] Reliable Kafka Receiver can lose data if the blo...
...ck generator fails to store data.
The Reliable Kafka Receiver commits offsets only when events are actually stored, which ensures that on restart we will actually start where we left off. But if the failure happens in the store() call, and the block generator reports an error the receiver does not do anything and will continue reading from the current offset and not the last commit. This means that messages between the last commit and the current offset will be lost.
This PR retries the store call four times and then stops the receiver with an error message and the last exception that was received from the store.
Author: Hari Shreedharan <hshreedharan@apache.org>
Closes #3655 from harishreedharan/kafka-failure-fix and squashes the following commits:
5e2e7ad [Hari Shreedharan] [SPARK-4704][STREAMING] Reliable Kafka Receiver can lose data if the block generator fails to store data.
Diffstat (limited to 'docs/java-programming-guide.md')
0 files changed, 0 insertions, 0 deletions