Skip to content

Commit

Permalink
Include reference to twitter/chill in tuning docs
Browse files Browse the repository at this point in the history
Author: Andrew Ash <[email protected]>

Closes apache#647 from ash211/doc-tuning and squashes the following commits:

b87de0a [Andrew Ash] Include reference to twitter/chill in tuning docs
  • Loading branch information
ash211 authored and rxin committed Feb 25, 2014
1 parent 4d88030 commit a4f4fbc
Showing 1 changed file with 6 additions and 3 deletions.
9 changes: 6 additions & 3 deletions docs/tuning.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,10 @@ This setting configures the serializer used for not only shuffling data between
nodes but also when serializing RDDs to disk. The only reason Kryo is not the default is because of the custom
registration requirement, but we recommend trying it in any network-intensive application.

Finally, to register your classes with Kryo, create a public class that extends
Spark automatically includes Kryo serializers for the many commonly-used core Scala classes covered
in the AllScalaRegistrar from the [Twitter chill](https://github.com/twitter/chill) library.

To register your own custom classes with Kryo, create a public class that extends
[`org.apache.spark.serializer.KryoRegistrator`](api/core/index.html#org.apache.spark.serializer.KryoRegistrator) and set the
`spark.kryo.registrator` config property to point to it, as follows:

Expand Down Expand Up @@ -72,8 +75,8 @@ If your objects are large, you may also need to increase the `spark.kryoserializ
config property. The default is 2, but this value needs to be large enough to hold the *largest*
object you will serialize.

Finally, if you don't register your classes, Kryo will still work, but it will have to store the
full class name with each object, which is wasteful.
Finally, if you don't register your custom classes, Kryo will still work, but it will have to store
the full class name with each object, which is wasteful.

# Memory Tuning

Expand Down

0 comments on commit a4f4fbc

Please sign in to comment.