diff options
| author | cody koeninger <cody@koeninger.org> | 2016-11-03 14:43:25 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2016-11-03 14:43:25 -0700 |
| commit | 67659c9afaeb2289e56fd87fafee953e8f050383 (patch) | |
| tree | 6a00e1c2e005a4d1d099e72b1c0fa665f6872328 /external/kafka-0-10-sql | |
| parent | 098e4ca9c7af61e64839a50c65be449749af6482 (diff) | |
| download | spark-67659c9afaeb2289e56fd87fafee953e8f050383.tar.gz spark-67659c9afaeb2289e56fd87fafee953e8f050383.tar.bz2 spark-67659c9afaeb2289e56fd87fafee953e8f050383.zip | |
[SPARK-18212][SS][KAFKA] increase executor poll timeout
## What changes were proposed in this pull request?
Increase poll timeout to try and address flaky test
## How was this patch tested?
Ran existing unit tests
Author: cody koeninger <cody@koeninger.org>
Closes #15737 from koeninger/SPARK-18212.
Diffstat (limited to 'external/kafka-0-10-sql')
| -rw-r--r-- | external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala | 5 |
1 files changed, 4 insertions, 1 deletions
diff --git a/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala b/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala index 61cba737d1..b21508cd7e 100644 --- a/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala +++ b/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala @@ -88,7 +88,10 @@ private[kafka010] case class KafkaSource( private val sc = sqlContext.sparkContext - private val pollTimeoutMs = sourceOptions.getOrElse("kafkaConsumer.pollTimeoutMs", "512").toLong + private val pollTimeoutMs = sourceOptions.getOrElse( + "kafkaConsumer.pollTimeoutMs", + sc.conf.getTimeAsMs("spark.network.timeout", "120s").toString + ).toLong private val maxOffsetFetchAttempts = sourceOptions.getOrElse("fetchOffset.numRetries", "3").toInt |