diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2015-04-01 16:06:11 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-04-01 16:06:11 -0700 |
| commit | f084c5de14eb10a6aba82a39e03e7877926ebb9e (patch) | |
| tree | 6a874a0f19171c3f6c68e4de07fa8ae7ac85c260 /network/common/pom.xml | |
| parent | fb25e8c7f45b4f96561e3f7434a0f4dfce8ddefe (diff) | |
| download | spark-f084c5de14eb10a6aba82a39e03e7877926ebb9e.tar.gz spark-f084c5de14eb10a6aba82a39e03e7877926ebb9e.tar.bz2 spark-f084c5de14eb10a6aba82a39e03e7877926ebb9e.zip | |
[SPARK-6578] [core] Fix thread-safety issue in outbound path of network library.
While the inbound path of a netty pipeline is thread-safe, the outbound
path is not. That means that multiple threads can compete to write messages
to the next stage of the pipeline.
The network library sometimes breaks a single RPC message into multiple
buffers internally to avoid copying data (see MessageEncoder). This can
result in the following scenario (where "FxBy" means "frame x, buffer y"):
T1 F1B1 F1B2
\ \
\ \
socket F1B1 F2B1 F1B2 F2B2
/ /
/ /
T2 F2B1 F2B2
And the frames now cannot be rebuilt on the receiving side because the
different messages have been mixed up on the wire.
The fix wraps these multi-buffer messages into a `FileRegion` object
so that these messages are written "atomically" to the next pipeline handler.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #5234 from vanzin/SPARK-6578 and squashes the following commits:
16b2d70 [Marcelo Vanzin] Forgot to update a type.
c9c2e4e [Marcelo Vanzin] Review comments: simplify some code.
9c888ac [Marcelo Vanzin] Small style nits.
8474bab [Marcelo Vanzin] Fix multiple calls to MessageWithHeader.transferTo().
e26509f [Marcelo Vanzin] Merge branch 'master' into SPARK-6578
c503f6c [Marcelo Vanzin] Implement a custom FileRegion instead of using locks.
84aa7ce [Marcelo Vanzin] Rename handler to the correct name.
432f3bd [Marcelo Vanzin] Remove unneeded method.
8d70e60 [Marcelo Vanzin] Fix thread-safety issue in outbound path of network library.
Diffstat (limited to 'network/common/pom.xml')
| -rw-r--r-- | network/common/pom.xml | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/network/common/pom.xml b/network/common/pom.xml index 7b51845206..22c738bde6 100644 --- a/network/common/pom.xml +++ b/network/common/pom.xml @@ -80,6 +80,11 @@ <artifactId>mockito-all</artifactId> <scope>test</scope> </dependency> + <dependency> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + <scope>test</scope> + </dependency> </dependencies> <build> |