diff options
| author | cody koeninger <cody@koeninger.org> | 2016-10-21 16:27:19 -0700 |
|---|---|---|
| committer | Shixiong Zhu <shixiong@databricks.com> | 2016-10-21 16:27:19 -0700 |
| commit | c9720b2195a465653690b3e221ce789142217b0d (patch) | |
| tree | 5300f2d721a6562a49fe71d64ef5ecf74262a5a5 /docs/streaming-kafka-0-10-integration.md | |
| parent | 268ccb9a48dfefc4d7bc85155e7e20a2dfe89307 (diff) | |
| download | spark-c9720b2195a465653690b3e221ce789142217b0d.tar.gz spark-c9720b2195a465653690b3e221ce789142217b0d.tar.bz2 spark-c9720b2195a465653690b3e221ce789142217b0d.zip | |
[STREAMING][KAFKA][DOC] clarify kafka settings needed for larger batches
## What changes were proposed in this pull request?
Minor doc change to mention kafka configuration for larger spark batches.
## How was this patch tested?
Doc change only, confirmed via jekyll.
The configuration issue was discussed / confirmed with users on the mailing list.
Author: cody koeninger <cody@koeninger.org>
Closes #15570 from koeninger/kafka-doc-heartbeat.
Diffstat (limited to 'docs/streaming-kafka-0-10-integration.md')
| -rw-r--r-- | docs/streaming-kafka-0-10-integration.md | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/docs/streaming-kafka-0-10-integration.md b/docs/streaming-kafka-0-10-integration.md index 456b845338..de95ea9013 100644 --- a/docs/streaming-kafka-0-10-integration.md +++ b/docs/streaming-kafka-0-10-integration.md @@ -48,6 +48,7 @@ Each item in the stream is a [ConsumerRecord](http://kafka.apache.org/0100/javad </div> For possible kafkaParams, see [Kafka consumer config docs](http://kafka.apache.org/documentation.html#newconsumerconfigs). +If your Spark batch duration is larger than the default Kafka heartbeat session timeout (30 seconds), increase heartbeat.interval.ms and session.timeout.ms appropriately. For batches larger than 5 minutes, this will require changing group.max.session.timeout.ms on the broker. Note that the example sets enable.auto.commit to false, for discussion see [Storing Offsets](streaming-kafka-0-10-integration.html#storing-offsets) below. ### LocationStrategies |