diff options
author | Sandy Ryza <sandy@cloudera.com> | 2014-10-21 21:53:09 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-10-21 21:53:09 -0700 |
commit | 6bb56faea8d238ea22c2de33db93b1b39f492b3a (patch) | |
tree | 4f172ee623cd64827b6c78d8459554cd395b1660 /docs | |
parent | 856b081729057f9da31a86e4bfa0dc0013492042 (diff) | |
download | spark-6bb56faea8d238ea22c2de33db93b1b39f492b3a.tar.gz spark-6bb56faea8d238ea22c2de33db93b1b39f492b3a.tar.bz2 spark-6bb56faea8d238ea22c2de33db93b1b39f492b3a.zip |
SPARK-1813. Add a utility to SparkConf that makes using Kryo really easy
Author: Sandy Ryza <sandy@cloudera.com>
Closes #789 from sryza/sandy-spark-1813 and squashes the following commits:
48b05e9 [Sandy Ryza] Simplify
b824932 [Sandy Ryza] Allow both spark.kryo.classesToRegister and spark.kryo.registrator at the same time
6a15bb7 [Sandy Ryza] Small fix
a2278c0 [Sandy Ryza] Respond to review comments
6ef592e [Sandy Ryza] SPARK-1813. Add a utility to SparkConf that makes using Kryo really easy
Diffstat (limited to 'docs')
-rw-r--r-- | docs/configuration.md | 15 | ||||
-rw-r--r-- | docs/tuning.md | 17 |
2 files changed, 15 insertions, 17 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 96fa1377ec..66738d3ca7 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -125,11 +125,22 @@ of the most common options to set are: </td> </tr> <tr> + <td><code>spark.kryo.classesToRegister</code></td> + <td>(none)</td> + <td> + If you use Kryo serialization, give a comma-separated list of custom class names to register + with Kryo. + See the <a href="tuning.html#data-serialization">tuning guide</a> for more details. + </td> +</tr> +<tr> <td><code>spark.kryo.registrator</code></td> <td>(none)</td> <td> - If you use Kryo serialization, set this class to register your custom classes with Kryo. - It should be set to a class that extends + If you use Kryo serialization, set this class to register your custom classes with Kryo. This + property is useful if you need to register your classes in a custom way, e.g. to specify a custom + field serializer. Otherwise <code>spark.kryo.classesToRegister</code> is simpler. It should be + set to a class that extends <a href="api/scala/index.html#org.apache.spark.serializer.KryoRegistrator"> <code>KryoRegistrator</code></a>. See the <a href="tuning.html#data-serialization">tuning guide</a> for more details. diff --git a/docs/tuning.md b/docs/tuning.md index 8fb2a0433b..9b5c9adac6 100644 --- a/docs/tuning.md +++ b/docs/tuning.md @@ -47,24 +47,11 @@ registration requirement, but we recommend trying it in any network-intensive ap Spark automatically includes Kryo serializers for the many commonly-used core Scala classes covered in the AllScalaRegistrar from the [Twitter chill](https://github.com/twitter/chill) library. -To register your own custom classes with Kryo, create a public class that extends -[`org.apache.spark.serializer.KryoRegistrator`](api/scala/index.html#org.apache.spark.serializer.KryoRegistrator) and set the -`spark.kryo.registrator` config property to point to it, as follows: +To register your own custom classes with Kryo, use the `registerKryoClasses` method. {% highlight scala %} -import com.esotericsoftware.kryo.Kryo -import org.apache.spark.serializer.KryoRegistrator - -class MyRegistrator extends KryoRegistrator { - override def registerClasses(kryo: Kryo) { - kryo.register(classOf[MyClass1]) - kryo.register(classOf[MyClass2]) - } -} - val conf = new SparkConf().setMaster(...).setAppName(...) -conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") -conf.set("spark.kryo.registrator", "mypackage.MyRegistrator") +conf.registerKryoClasses(Seq(classOf[MyClass1], classOf[MyClass2])) val sc = new SparkContext(conf) {% endhighlight %} |