aboutsummaryrefslogtreecommitdiff
path: root/core
diff options
context:
space:
mode:
authorAnkur Dave <ankurdave@gmail.com>2014-07-23 20:11:28 -0700
committerReynold Xin <rxin@apache.org>2014-07-23 20:11:28 -0700
commit2d25e34814f81f288587f3277324cb655a5fb38d (patch)
tree0969341520725401d4d7309135b16ba5a499461e /core
parent60f0ae3d87c84fd96e1f4d0abf5be1f51870e7ab (diff)
downloadspark-2d25e34814f81f288587f3277324cb655a5fb38d.tar.gz
spark-2d25e34814f81f288587f3277324cb655a5fb38d.tar.bz2
spark-2d25e34814f81f288587f3277324cb655a5fb38d.zip
Replace RoutingTableMessage with pair
RoutingTableMessage was used to construct routing tables to enable joining VertexRDDs with partitioned edges. It stored three elements: the destination vertex ID, the source edge partition, and a byte specifying the position in which the edge partition referenced the vertex to enable join elimination. However, this was incompatible with sort-based shuffle (SPARK-2045). It was also slightly wasteful, because partition IDs are usually much smaller than 2^32, though this was mitigated by a custom serializer that used variable-length encoding. This commit replaces RoutingTableMessage with a pair of (VertexId, Int) where the Int encodes both the source partition ID (in the lower 30 bits) and the position (in the top 2 bits). Author: Ankur Dave <ankurdave@gmail.com> Closes #1553 from ankurdave/remove-RoutingTableMessage and squashes the following commits: 697e17b [Ankur Dave] Replace RoutingTableMessage with pair
Diffstat (limited to 'core')
0 files changed, 0 insertions, 0 deletions