diff options
| author | Dhruve Ashar <dhruveashar@gmail.com> | 2016-06-17 15:44:33 -0500 |
|---|---|---|
| committer | Tom Graves <tgraves@yahoo-inc.com> | 2016-06-17 15:44:33 -0500 |
| commit | 298c4ae81520b6b39230a6b0bf733c2b7caea627 (patch) | |
| tree | 357829c88c8e8c656ce79403e097f7ccd508a85f /common | |
| parent | c8809db5a5ae4111e193907ac35929a906ddac3e (diff) | |
| download | spark-298c4ae81520b6b39230a6b0bf733c2b7caea627.tar.gz spark-298c4ae81520b6b39230a6b0bf733c2b7caea627.tar.bz2 spark-298c4ae81520b6b39230a6b0bf733c2b7caea627.zip | |
[SPARK-16018][SHUFFLE] Shade netty to load shuffle jar in Nodemanger
## What changes were proposed in this pull request?
Shade the netty.io namespace so that we can use it in shuffle independent of the dependencies being pulled by hadoop jars.
## How was this patch tested?
Ran a decent job involving shuffle write/read and tested the new spark-x-yarn-shuffle jar. After shading netty.io namespace, the nodemanager loads and shuffle job completes successfully.
Author: Dhruve Ashar <dhruveashar@gmail.com>
Closes #13739 from dhruve/bug/SPARK-16018.
Diffstat (limited to 'common')
| -rw-r--r-- | common/network-yarn/pom.xml | 7 |
1 files changed, 7 insertions, 0 deletions
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml index 1fd3af2e6e..3b7ffe8277 100644 --- a/common/network-yarn/pom.xml +++ b/common/network-yarn/pom.xml @@ -96,6 +96,13 @@ <include>com.fasterxml.jackson.**</include> </includes> </relocation> + <relocation> + <pattern>io.netty</pattern> + <shadedPattern>${spark.shade.packageName}.io.netty</shadedPattern> + <includes> + <include>io.netty.**</include> + </includes> + </relocation> </relocations> </configuration> <executions> |