aboutsummaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorAndrew Ash <andrew@andrewash.com>2014-02-24 21:13:38 -0800
committerReynold Xin <rxin@apache.org>2014-02-24 21:13:38 -0800
commita4f4fbc8fa5886a8c6ee58ee614de0cc6e67dcd7 (patch)
tree00ec069ac2e4b145042ec808dd00b80eca324a72 /docs
parent4d880304867b55a4f2138617b30600b7fa013b14 (diff)
downloadspark-a4f4fbc8fa5886a8c6ee58ee614de0cc6e67dcd7.tar.gz
spark-a4f4fbc8fa5886a8c6ee58ee614de0cc6e67dcd7.tar.bz2
spark-a4f4fbc8fa5886a8c6ee58ee614de0cc6e67dcd7.zip
Include reference to twitter/chill in tuning docs
Author: Andrew Ash <andrew@andrewash.com> Closes #647 from ash211/doc-tuning and squashes the following commits: b87de0a [Andrew Ash] Include reference to twitter/chill in tuning docs
Diffstat (limited to 'docs')
-rw-r--r--docs/tuning.md9
1 files changed, 6 insertions, 3 deletions
diff --git a/docs/tuning.md b/docs/tuning.md
index 6b010aed61..704778681c 100644
--- a/docs/tuning.md
+++ b/docs/tuning.md
@@ -44,7 +44,10 @@ This setting configures the serializer used for not only shuffling data between
nodes but also when serializing RDDs to disk. The only reason Kryo is not the default is because of the custom
registration requirement, but we recommend trying it in any network-intensive application.
-Finally, to register your classes with Kryo, create a public class that extends
+Spark automatically includes Kryo serializers for the many commonly-used core Scala classes covered
+in the AllScalaRegistrar from the [Twitter chill](https://github.com/twitter/chill) library.
+
+To register your own custom classes with Kryo, create a public class that extends
[`org.apache.spark.serializer.KryoRegistrator`](api/core/index.html#org.apache.spark.serializer.KryoRegistrator) and set the
`spark.kryo.registrator` config property to point to it, as follows:
@@ -72,8 +75,8 @@ If your objects are large, you may also need to increase the `spark.kryoserializ
config property. The default is 2, but this value needs to be large enough to hold the *largest*
object you will serialize.
-Finally, if you don't register your classes, Kryo will still work, but it will have to store the
-full class name with each object, which is wasteful.
+Finally, if you don't register your custom classes, Kryo will still work, but it will have to store
+the full class name with each object, which is wasteful.
# Memory Tuning