diff options
| author | Ankur Dave <ankurdave@gmail.com> | 2014-07-22 22:18:30 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@apache.org> | 2014-07-22 22:18:30 -0700 |
| commit | 6c2be93f081f33e9e97e1231b0084a6a0eb4fa22 (patch) | |
| tree | 351799634cd34b39d2d0263395fbfb00d6434d56 /README.md | |
| parent | 02e45729472a22a31629cdd17dc3836ba6810189 (diff) | |
| download | spark-6c2be93f081f33e9e97e1231b0084a6a0eb4fa22.tar.gz spark-6c2be93f081f33e9e97e1231b0084a6a0eb4fa22.tar.bz2 spark-6c2be93f081f33e9e97e1231b0084a6a0eb4fa22.zip | |
Remove GraphX MessageToPartition for compatibility with sort-based shuffle
MessageToPartition was used in `Graph#partitionBy`. Unlike a Tuple2, it marked the key as transient to avoid sending it over the network. However, it was incompatible with sort-based shuffle (SPARK-2045) and represented only a minor optimization: for partitionBy, it improved performance by 6.3% (30.4 s to 28.5 s) and reduced communication by 5.6% (114.2 MB to 107.8 MB).
Author: Ankur Dave <ankurdave@gmail.com>
Closes #1537 from ankurdave/remove-MessageToPartition and squashes the following commits:
f9d0054 [Ankur Dave] Remove MessageToPartition
ab71364 [Ankur Dave] Remove unused VertexBroadcastMsg
Diffstat (limited to 'README.md')
0 files changed, 0 insertions, 0 deletions