diff options
| author | Issac Buenrostro <buenrostro@ooyala.com> | 2014-07-10 16:01:08 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2014-07-10 16:01:08 -0700 |
| commit | 2dd67248503306bb08946b1796821e9f9ed4d00e (patch) | |
| tree | a39be5352c3e2ca6467fbca6bee6430ce84974d4 /docs/configuration.md | |
| parent | 40a8fef4e6619b4ea10a4ec9026260649ce5ae73 (diff) | |
| download | spark-2dd67248503306bb08946b1796821e9f9ed4d00e.tar.gz spark-2dd67248503306bb08946b1796821e9f9ed4d00e.tar.bz2 spark-2dd67248503306bb08946b1796821e9f9ed4d00e.zip | |
[SPARK-1341] [Streaming] Throttle BlockGenerator to limit rate of data consumption.
Author: Issac Buenrostro <buenrostro@ooyala.com>
Closes #945 from ibuenros/SPARK-1341-throttle and squashes the following commits:
5514916 [Issac Buenrostro] Formatting changes, added documentation for streaming throttling, stricter unit tests for throttling.
62f395f [Issac Buenrostro] Add comments and license to streaming RateLimiter.scala
7066438 [Issac Buenrostro] Moved throttle code to RateLimiter class, smoother pushing when throttling active
ccafe09 [Issac Buenrostro] Throttle BlockGenerator to limit rate of data consumption.
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 9 |
1 files changed, 9 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index b84104cc7e..0aea23ab59 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -774,6 +774,15 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.streaming.receiver.maxRate</code></td> + <td>infinite</td> + <td> + Maximum rate (per second) at which each receiver will push data into blocks. Effectively, + each stream will consume at most this number of records per second. + Setting this configuration to 0 or a negative number will put no limit on the rate. + </td> +</tr> +<tr> <td><code>spark.streaming.unpersist</code></td> <td>true</td> <td> |