diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2015-11-23 13:54:19 -0800 |
|---|---|---|
| committer | Marcelo Vanzin <vanzin@cloudera.com> | 2015-11-23 13:54:19 -0800 |
| commit | c2467dadae8ce44010a912ee91c429310f8add65 (patch) | |
| tree | 13f1eb5a8e2e2348a6a0b1296a720b8ef626eaf9 /network | |
| parent | 7cfa4c6bc36d97e459d4adee7b03d537d63c337e (diff) | |
| download | spark-c2467dadae8ce44010a912ee91c429310f8add65.tar.gz spark-c2467dadae8ce44010a912ee91c429310f8add65.tar.bz2 spark-c2467dadae8ce44010a912ee91c429310f8add65.zip | |
[SPARK-11140][CORE] Transfer files using network lib when using NettyRpcEnv.
This change abstracts the code that serves jars / files to executors so that
each RpcEnv can have its own implementation; the akka version uses the existing
HTTP-based file serving mechanism, while the netty versions uses the new
stream support added to the network lib, which makes file transfers benefit
from the easier security configuration of the network library, and should also
reduce overhead overall.
The change includes a small fix to TransportChannelHandler so that it propagates
user events to downstream handlers.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #9530 from vanzin/SPARK-11140.
Diffstat (limited to 'network')
2 files changed, 5 insertions, 2 deletions
diff --git a/network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java b/network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java index 659c47160c..61bafc8380 100644 --- a/network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java +++ b/network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java @@ -170,8 +170,10 @@ public class TransportClientFactory implements Closeable { } /** - * Create a completely new {@link TransportClient} to the given remote host / port - * But this connection is not pooled. + * Create a completely new {@link TransportClient} to the given remote host / port. + * This connection is not pooled. + * + * As with {@link #createClient(String, int)}, this method is blocking. */ public TransportClient createUnmanagedClient(String remoteHost, int remotePort) throws IOException { diff --git a/network/common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java b/network/common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java index 29d688a675..3164e00679 100644 --- a/network/common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java +++ b/network/common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java @@ -138,6 +138,7 @@ public class TransportChannelHandler extends SimpleChannelInboundHandler<Message } } } + ctx.fireUserEventTriggered(evt); } public TransportResponseHandler getResponseHandler() { |