From 2dd67248503306bb08946b1796821e9f9ed4d00e Mon Sep 17 00:00:00 2001 From: Issac Buenrostro Date: Thu, 10 Jul 2014 16:01:08 -0700 Subject: [SPARK-1341] [Streaming] Throttle BlockGenerator to limit rate of data consumption. Author: Issac Buenrostro Closes #945 from ibuenros/SPARK-1341-throttle and squashes the following commits: 5514916 [Issac Buenrostro] Formatting changes, added documentation for streaming throttling, stricter unit tests for throttling. 62f395f [Issac Buenrostro] Add comments and license to streaming RateLimiter.scala 7066438 [Issac Buenrostro] Moved throttle code to RateLimiter class, smoother pushing when throttling active ccafe09 [Issac Buenrostro] Throttle BlockGenerator to limit rate of data consumption. --- docs/configuration.md | 9 +++++++++ 1 file changed, 9 insertions(+) (limited to 'docs') diff --git a/docs/configuration.md b/docs/configuration.md index b84104cc7e..0aea23ab59 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -773,6 +773,15 @@ Apart from these, the following properties are also available, and may be useful into blocks of data before storing them in Spark. + + spark.streaming.receiver.maxRate + infinite + + Maximum rate (per second) at which each receiver will push data into blocks. Effectively, + each stream will consume at most this number of records per second. + Setting this configuration to 0 or a negative number will put no limit on the rate. + + spark.streaming.unpersist true -- cgit v1.2.3