diff options
author | Ankur Dave <ankurdave@gmail.com> | 2014-07-23 20:11:28 -0700 |
---|---|---|
committer | Reynold Xin <rxin@apache.org> | 2014-07-23 20:11:28 -0700 |
commit | 2d25e34814f81f288587f3277324cb655a5fb38d (patch) | |
tree | 0969341520725401d4d7309135b16ba5a499461e /docs/java-programming-guide.md | |
parent | 60f0ae3d87c84fd96e1f4d0abf5be1f51870e7ab (diff) | |
download | spark-2d25e34814f81f288587f3277324cb655a5fb38d.tar.gz spark-2d25e34814f81f288587f3277324cb655a5fb38d.tar.bz2 spark-2d25e34814f81f288587f3277324cb655a5fb38d.zip |
Replace RoutingTableMessage with pair
RoutingTableMessage was used to construct routing tables to enable
joining VertexRDDs with partitioned edges. It stored three elements: the
destination vertex ID, the source edge partition, and a byte specifying
the position in which the edge partition referenced the vertex to enable
join elimination.
However, this was incompatible with sort-based shuffle (SPARK-2045). It
was also slightly wasteful, because partition IDs are usually much
smaller than 2^32, though this was mitigated by a custom serializer that
used variable-length encoding.
This commit replaces RoutingTableMessage with a pair of (VertexId, Int)
where the Int encodes both the source partition ID (in the lower 30
bits) and the position (in the top 2 bits).
Author: Ankur Dave <ankurdave@gmail.com>
Closes #1553 from ankurdave/remove-RoutingTableMessage and squashes the following commits:
697e17b [Ankur Dave] Replace RoutingTableMessage with pair
Diffstat (limited to 'docs/java-programming-guide.md')
0 files changed, 0 insertions, 0 deletions