diff options
| author | Liwei Lin <lwlin7@gmail.com> | 2017-02-12 23:00:22 -0800 |
|---|---|---|
| committer | Shixiong Zhu <shixiong@databricks.com> | 2017-02-12 23:00:22 -0800 |
| commit | 2bdbc87052389ff69404347fbc69457132dbcafd (patch) | |
| tree | 7e2da297c5ed55da43abd32d4d49e048d47f3ab2 /examples | |
| parent | bc0a0e6392c4e729d8f0e4caffc0bd05adb0d950 (diff) | |
| download | spark-2bdbc87052389ff69404347fbc69457132dbcafd.tar.gz spark-2bdbc87052389ff69404347fbc69457132dbcafd.tar.bz2 spark-2bdbc87052389ff69404347fbc69457132dbcafd.zip | |
[SPARK-19564][SPARK-19559][SS][KAFKA] KafkaOffsetReader's consumers should not be in the same group
## What changes were proposed in this pull request?
In `KafkaOffsetReader`, when error occurs, we abort the existing consumer and create a new consumer. In our current implementation, the first consumer and the second consumer would be in the same group (which leads to SPARK-19559), **_violating our intention of the two consumers not being in the same group._**
The cause is that, in our current implementation, the first consumer is created before `groupId` and `nextId` are initialized in the constructor. Then even if `groupId` and `nextId` are increased during the creation of that first consumer, `groupId` and `nextId` would still be initialized to default values in the constructor for the second consumer.
We should make sure that `groupId` and `nextId` are initialized before any consumer is created.
## How was this patch tested?
Ran 100 times of `KafkaSourceSuite`; all passed
Author: Liwei Lin <lwlin7@gmail.com>
Closes #16902 from lw-lin/SPARK-19564-.
Diffstat (limited to 'examples')
0 files changed, 0 insertions, 0 deletions