diff options
author | Josh Rosen <joshrosen@databricks.com> | 2015-05-06 10:52:55 -0700 |
---|---|---|
committer | Josh Rosen <joshrosen@databricks.com> | 2015-05-06 10:53:19 -0700 |
commit | d651e28383a590f260be62be91f1e0770a21267a (patch) | |
tree | f07c1611f91dfa16e8edc9121f09361fbd5555d6 /repl/scala-2.11/src | |
parent | b521a3b0308a08fb2ce79b03fc7c94f3131c2e4b (diff) | |
download | spark-d651e28383a590f260be62be91f1e0770a21267a.tar.gz spark-d651e28383a590f260be62be91f1e0770a21267a.tar.bz2 spark-d651e28383a590f260be62be91f1e0770a21267a.zip |
[SPARK-7311] Introduce internal Serializer API for determining if serializers support object relocation
This patch extends the `Serializer` interface with a new `Private` API which allows serializers to indicate whether they support relocation of serialized objects in serializer stream output.
This relocatibilty property is described in more detail in `Serializer.scala`, but in a nutshell a serializer supports relocation if reordering the bytes of serialized objects in serialization stream output is equivalent to having re-ordered those elements prior to serializing them. The optimized shuffle path introduced in #4450 and #5868 both rely on serializers having this property; this patch just centralizes the logic for determining whether a serializer has this property. I also added tests and comments clarifying when this works for KryoSerializer.
This change allows the optimizations in #4450 to be applied for shuffles that use `SqlSerializer2`.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #5924 from JoshRosen/SPARK-7311 and squashes the following commits:
50a68ca [Josh Rosen] Address minor nits
0a7ebd7 [Josh Rosen] Clarify reason why SqlSerializer2 supports this serializer
123b992 [Josh Rosen] Cleanup for submitting as standalone patch.
4aa61b2 [Josh Rosen] Add missing newline
2c1233a [Josh Rosen] Small refactoring of SerializerPropertiesSuite to enable test re-use:
0ba75e6 [Josh Rosen] Add tests for serializer relocation property.
450fa21 [Josh Rosen] Back out accidental log4j.properties change
86d4dcd [Josh Rosen] Flag that SparkSqlSerializer2 supports relocation
b9624ee [Josh Rosen] Expand serializer API and use new function to help control when new UnsafeShuffle path is used.
(cherry picked from commit 002c12384d6ecebbb3e7fc853dbdfbc5aaa3d6a6)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
Diffstat (limited to 'repl/scala-2.11/src')
0 files changed, 0 insertions, 0 deletions