diff options
author | Tathagata Das <tathagata.das1565@gmail.com> | 2015-09-08 14:54:43 -0700 |
---|---|---|
committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-09-08 14:54:43 -0700 |
commit | 52b24a602ad615a7f6aa427aefb1c7444c05d298 (patch) | |
tree | 8c64796dda119ac73ed79d05cf8b028b05e4158c /docs/configuration.md | |
parent | e6f8d3686016a305a747c5bcc85f46fd4c0cbe83 (diff) | |
download | spark-52b24a602ad615a7f6aa427aefb1c7444c05d298.tar.gz spark-52b24a602ad615a7f6aa427aefb1c7444c05d298.tar.bz2 spark-52b24a602ad615a7f6aa427aefb1c7444c05d298.zip |
[SPARK-10492] [STREAMING] [DOCUMENTATION] Update Streaming documentation about rate limiting and backpressure
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #8656 from tdas/SPARK-10492 and squashes the following commits:
986cdd6 [Tathagata Das] Added information on backpressure
Diffstat (limited to 'docs/configuration.md')
-rw-r--r-- | docs/configuration.md | 13 |
1 files changed, 13 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index a2cc7a37e2..e287591f3f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1434,6 +1434,19 @@ Apart from these, the following properties are also available, and may be useful <table class="table"> <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr> <tr> + <td><code>spark.streaming.backpressure.enabled</code></td> + <td>false</td> + <td> + Enables or disables Spark Streaming's internal backpressure mechanism (since 1.5). + This enables the Spark Streaming to control the receiving rate based on the + current batch scheduling delays and processing times so that the system receives + only as fast as the system can process. Internally, this dynamically sets the + maximum receiving rate of receivers. This rate is upper bounded by the values + `spark.streaming.receiver.maxRate` and `spark.streaming.kafka.maxRatePerPartition` + if they are set (see below). + </td> +</tr> +<tr> <td><code>spark.streaming.blockInterval</code></td> <td>200ms</td> <td> |