diff options
| author | cody koeninger <cody@koeninger.org> | 2016-10-27 10:30:59 -0700 |
|---|---|---|
| committer | Shixiong Zhu <shixiong@databricks.com> | 2016-10-27 10:30:59 -0700 |
| commit | 104232580528c097a284d753adb5795f6de8b0a5 (patch) | |
| tree | 3c7147652132b67b0f9bf4e9fe83f602c6c35127 /docs | |
| parent | 701a9d361b3045a25c42b3c0e44e7755d45ff78c (diff) | |
| download | spark-104232580528c097a284d753adb5795f6de8b0a5.tar.gz spark-104232580528c097a284d753adb5795f6de8b0a5.tar.bz2 spark-104232580528c097a284d753adb5795f6de8b0a5.zip | |
[SPARK-17813][SQL][KAFKA] Maximum data per trigger
## What changes were proposed in this pull request?
maxOffsetsPerTrigger option for rate limiting, proportionally based on volume of different topicpartitions.
## How was this patch tested?
Added unit test
Author: cody koeninger <cody@koeninger.org>
Closes #15527 from koeninger/SPARK-17813.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/structured-streaming-kafka-integration.md | 6 |
1 files changed, 6 insertions, 0 deletions
diff --git a/docs/structured-streaming-kafka-integration.md b/docs/structured-streaming-kafka-integration.md index e851f210c9..a6c3b3a902 100644 --- a/docs/structured-streaming-kafka-integration.md +++ b/docs/structured-streaming-kafka-integration.md @@ -221,6 +221,12 @@ The following configurations are optional: <td>10</td> <td>milliseconds to wait before retrying to fetch Kafka offsets</td> </tr> +<tr> + <td>maxOffsetsPerTrigger</td> + <td>long</td> + <td>none</td> + <td>Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume.</td> +</tr> </table> Kafka's own configurations can be set via `DataStreamReader.option` with `kafka.` prefix, e.g, |