diff options
author | Lianhui Wang <lianhuiwang09@gmail.com> | 2015-11-10 10:40:08 -0800 |
---|---|---|
committer | Marcelo Vanzin <vanzin@cloudera.com> | 2015-11-10 10:40:08 -0800 |
commit | 6e5fc37883ed81c3ee2338145a48de3036d19399 (patch) | |
tree | 67c33f795a8764706e7598f4afcae3eb00f42c41 /core/src/main | |
parent | 689386b1c60997e4505749915f7005a52c207de2 (diff) | |
download | spark-6e5fc37883ed81c3ee2338145a48de3036d19399.tar.gz spark-6e5fc37883ed81c3ee2338145a48de3036d19399.tar.bz2 spark-6e5fc37883ed81c3ee2338145a48de3036d19399.zip |
[SPARK-11252][NETWORK] ShuffleClient should release connection after fetching blocks had been completed for external shuffle
with yarn's external shuffle, ExternalShuffleClient of executors reserve its connections for yarn's NodeManager until application has been completed. so it will make NodeManager and executors have many socket connections.
in order to reduce network pressure of NodeManager's shuffleService, after registerWithShuffleServer or fetchBlocks have been completed in ExternalShuffleClient, connection for NM's shuffleService needs to be closed.andrewor14 rxin vanzin
Author: Lianhui Wang <lianhuiwang09@gmail.com>
Closes #9227 from lianhuiwang/spark-11252.
Diffstat (limited to 'core/src/main')
-rw-r--r-- | core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala b/core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala index 6840a3ae83..a039d543c3 100644 --- a/core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala +++ b/core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala @@ -47,7 +47,8 @@ class ExternalShuffleService(sparkConf: SparkConf, securityManager: SecurityMana private val transportConf = SparkTransportConf.fromSparkConf(sparkConf, numUsableCores = 0) private val blockHandler = newShuffleBlockHandler(transportConf) - private val transportContext: TransportContext = new TransportContext(transportConf, blockHandler) + private val transportContext: TransportContext = + new TransportContext(transportConf, blockHandler, true) private var server: TransportServer = _ |