diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2015-11-23 13:54:19 -0800 |
|---|---|---|
| committer | Marcelo Vanzin <vanzin@cloudera.com> | 2015-11-23 13:54:19 -0800 |
| commit | c2467dadae8ce44010a912ee91c429310f8add65 (patch) | |
| tree | 13f1eb5a8e2e2348a6a0b1296a720b8ef626eaf9 /launcher | |
| parent | 7cfa4c6bc36d97e459d4adee7b03d537d63c337e (diff) | |
| download | spark-c2467dadae8ce44010a912ee91c429310f8add65.tar.gz spark-c2467dadae8ce44010a912ee91c429310f8add65.tar.bz2 spark-c2467dadae8ce44010a912ee91c429310f8add65.zip | |
[SPARK-11140][CORE] Transfer files using network lib when using NettyRpcEnv.
This change abstracts the code that serves jars / files to executors so that
each RpcEnv can have its own implementation; the akka version uses the existing
HTTP-based file serving mechanism, while the netty versions uses the new
stream support added to the network lib, which makes file transfers benefit
from the easier security configuration of the network library, and should also
reduce overhead overall.
The change includes a small fix to TransportChannelHandler so that it propagates
user events to downstream handlers.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #9530 from vanzin/SPARK-11140.
Diffstat (limited to 'launcher')
| -rw-r--r-- | launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java b/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java index 3ee6bd92e4..55fe156cf6 100644 --- a/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java +++ b/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java @@ -148,7 +148,7 @@ abstract class AbstractCommandBuilder { String scala = getScalaVersion(); List<String> projects = Arrays.asList("core", "repl", "mllib", "bagel", "graphx", "streaming", "tools", "sql/catalyst", "sql/core", "sql/hive", "sql/hive-thriftserver", - "yarn", "launcher"); + "yarn", "launcher", "network/common", "network/shuffle", "network/yarn"); if (prependClasses) { if (!isTesting) { System.err.println( |