aboutsummaryrefslogtreecommitdiff
path: root/docs/configuration.md
diff options
context:
space:
mode:
authorTathagata Das <tathagata.das1565@gmail.com>2015-09-08 14:54:43 -0700
committerTathagata Das <tathagata.das1565@gmail.com>2015-09-08 14:54:43 -0700
commit52b24a602ad615a7f6aa427aefb1c7444c05d298 (patch)
tree8c64796dda119ac73ed79d05cf8b028b05e4158c /docs/configuration.md
parente6f8d3686016a305a747c5bcc85f46fd4c0cbe83 (diff)
downloadspark-52b24a602ad615a7f6aa427aefb1c7444c05d298.tar.gz
spark-52b24a602ad615a7f6aa427aefb1c7444c05d298.tar.bz2
spark-52b24a602ad615a7f6aa427aefb1c7444c05d298.zip
[SPARK-10492] [STREAMING] [DOCUMENTATION] Update Streaming documentation about rate limiting and backpressure
Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #8656 from tdas/SPARK-10492 and squashes the following commits: 986cdd6 [Tathagata Das] Added information on backpressure
Diffstat (limited to 'docs/configuration.md')
-rw-r--r--docs/configuration.md13
1 files changed, 13 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md
index a2cc7a37e2..e287591f3f 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1434,6 +1434,19 @@ Apart from these, the following properties are also available, and may be useful
<table class="table">
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr>
+ <td><code>spark.streaming.backpressure.enabled</code></td>
+ <td>false</td>
+ <td>
+ Enables or disables Spark Streaming's internal backpressure mechanism (since 1.5).
+ This enables the Spark Streaming to control the receiving rate based on the
+ current batch scheduling delays and processing times so that the system receives
+ only as fast as the system can process. Internally, this dynamically sets the
+ maximum receiving rate of receivers. This rate is upper bounded by the values
+ `spark.streaming.receiver.maxRate` and `spark.streaming.kafka.maxRatePerPartition`
+ if they are set (see below).
+ </td>
+</tr>
+<tr>
<td><code>spark.streaming.blockInterval</code></td>
<td>200ms</td>
<td>