diff options
| author | Sanket <schintap@untilservice-lm> | 2016-02-11 22:40:00 -0800 |
|---|---|---|
| committer | Shixiong Zhu <shixiong@databricks.com> | 2016-02-11 22:40:00 -0800 |
| commit | 894921d813a259f2f266fde7d86d2ecb5a0af24b (patch) | |
| tree | fe70e1630e03c0ada42d486a5212628a24ce6c2b /docs/configuration.md | |
| parent | a2c7dcf61f33fa1897c950d2d905651103c170ea (diff) | |
| download | spark-894921d813a259f2f266fde7d86d2ecb5a0af24b.tar.gz spark-894921d813a259f2f266fde7d86d2ecb5a0af24b.tar.bz2 spark-894921d813a259f2f266fde7d86d2ecb5a0af24b.zip | |
[SPARK-6166] Limit number of in flight outbound requests
This JIRA is related to
https://github.com/apache/spark/pull/5852
Had to do some minor rework and test to make sure it
works with current version of spark.
Author: Sanket <schintap@untilservice-lm>
Closes #10838 from redsanket/limit-outbound-connections.
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 10 |
1 files changed, 10 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index dd2cde8194..0dbfe3b079 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -392,6 +392,16 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.reducer.maxReqsInFlight</code></td> + <td>Int.MaxValue</td> + <td> + This configuration limits the number of remote requests to fetch blocks at any given point. + When the number of hosts in the cluster increase, it might lead to very large number + of in-bound connections to one or more nodes, causing the workers to fail under load. + By allowing it to limit the number of fetch requests, this scenario can be mitigated. + </td> +</tr> +<tr> <td><code>spark.shuffle.compress</code></td> <td>true</td> <td> |