diff options
| author | Reynold Xin <rxin@databricks.com> | 2014-12-09 17:49:59 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2014-12-09 17:50:11 -0800 |
| commit | 441ec3451730c7ae3dbef8952e313071d6147ab6 (patch) | |
| tree | b9f64470f2ba45ad0d99c8f102775846981a779a /LICENSE | |
| parent | b0d64e57255e5ca545c90f18bd9d10a07ae43759 (diff) | |
| download | spark-441ec3451730c7ae3dbef8952e313071d6147ab6.tar.gz spark-441ec3451730c7ae3dbef8952e313071d6147ab6.tar.bz2 spark-441ec3451730c7ae3dbef8952e313071d6147ab6.zip | |
[SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty.
It's been reported that when the number of disks is large and the number of nodes is small, Netty network throughput is low compared with NIO. We suspect the problem is that only a small number of disks are utilized to serve shuffle files at any given point, due to connection reuse. This patch adds a new config parameter to specify the number of concurrent connections between two peer nodes, default to 2.
Author: Reynold Xin <rxin@databricks.com>
Closes #3625 from rxin/SPARK-4740 and squashes the following commits:
ad4241a [Reynold Xin] Updated javadoc.
f33c72b [Reynold Xin] Code review feedback.
0fefabb [Reynold Xin] Use double check in synchronization.
41dfcb2 [Reynold Xin] Added test case.
9076b4a [Reynold Xin] Fixed two NPEs.
3e1306c [Reynold Xin] Minor style fix.
4f21673 [Reynold Xin] [SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty.
(cherry picked from commit 2b9b72682e587909a84d3ace214c22cec830eeaf)
Signed-off-by: Reynold Xin <rxin@databricks.com>
Diffstat (limited to 'LICENSE')
0 files changed, 0 insertions, 0 deletions