diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2015-11-02 10:26:36 -0800 |
|---|---|---|
| committer | Marcelo Vanzin <vanzin@cloudera.com> | 2015-11-02 10:26:36 -0800 |
| commit | 71d1c907dec446db566b19f912159fd8f46deb7d (patch) | |
| tree | 8201803d422933421b5af731d61ef7dcd54ca6da /network | |
| parent | a930e624eb9feb0f7d37d99dcb8178feb9c0f177 (diff) | |
| download | spark-71d1c907dec446db566b19f912159fd8f46deb7d.tar.gz spark-71d1c907dec446db566b19f912159fd8f46deb7d.tar.bz2 spark-71d1c907dec446db566b19f912159fd8f46deb7d.zip | |
[SPARK-10997][CORE] Add "client mode" to netty rpc env.
"Client mode" means the RPC env will not listen for incoming connections.
This allows certain processes in the Spark stack (such as Executors or
tha YARN client-mode AM) to act as pure clients when using the netty-based
RPC backend, reducing the number of sockets needed by the app and also the
number of open ports.
Client connections are also preferred when endpoints that actually have
a listening socket are involved; so, for example, if a Worker connects
to a Master and the Master needs to send a message to a Worker endpoint,
that client connection will be used, even though the Worker is also
listening for incoming connections.
With this change, the workaround for SPARK-10987 isn't necessary anymore, and
is removed. The AM connects to the driver in "client mode", and that connection
is used for all driver <-> AM communication, and so the AM is properly notified
when the connection goes down.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #9210 from vanzin/SPARK-10997.
Diffstat (limited to 'network')
| -rw-r--r-- | network/yarn/pom.xml | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/network/yarn/pom.xml b/network/yarn/pom.xml index 541ed9a8d0..e2360eff5c 100644 --- a/network/yarn/pom.xml +++ b/network/yarn/pom.xml @@ -54,6 +54,11 @@ <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> </dependency> + <dependency> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + <scope>provided</scope> + </dependency> </dependencies> <build> |