diff options
| author | Sandeep Singh <sandeep@techaddict.me> | 2016-10-22 12:03:37 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-10-22 12:03:37 -0700 |
| commit | bc167a2a53f5a795d089e8a884569b1b3e2cd439 (patch) | |
| tree | 2b71829d0cf5f6737686533fef5d1dccf040c950 /docs | |
| parent | 4f1dcd3dce270268b42fbe59409790364fa5c5df (diff) | |
| download | spark-bc167a2a53f5a795d089e8a884569b1b3e2cd439.tar.gz spark-bc167a2a53f5a795d089e8a884569b1b3e2cd439.tar.bz2 spark-bc167a2a53f5a795d089e8a884569b1b3e2cd439.zip | |
[SPARK-928][CORE] Add support for Unsafe-based serializer in Kryo
## What changes were proposed in this pull request?
Now since we have migrated to Kryo-3.0.0 in https://issues.apache.org/jira/browse/SPARK-11416, we can gives users option to use unsafe SerDer. It can turned by setting `spark.kryo.useUnsafe` to `true`
## How was this patch tested?
Ran existing tests
```
Benchmark Kryo Unsafe vs safe Serialization: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
basicTypes: Int unsafe:true 160 / 178 98.5 10.1 1.0X
basicTypes: Long unsafe:true 210 / 218 74.9 13.4 0.8X
basicTypes: Float unsafe:true 203 / 213 77.5 12.9 0.8X
basicTypes: Double unsafe:true 226 / 235 69.5 14.4 0.7X
Array: Int unsafe:true 1087 / 1101 14.5 69.1 0.1X
Array: Long unsafe:true 2758 / 2844 5.7 175.4 0.1X
Array: Float unsafe:true 1511 / 1552 10.4 96.1 0.1X
Array: Double unsafe:true 2942 / 2972 5.3 187.0 0.1X
Map of string->Double unsafe:true 2645 / 2739 5.9 168.2 0.1X
basicTypes: Int unsafe:false 211 / 218 74.7 13.4 0.8X
basicTypes: Long unsafe:false 247 / 253 63.6 15.7 0.6X
basicTypes: Float unsafe:false 211 / 216 74.5 13.4 0.8X
basicTypes: Double unsafe:false 227 / 233 69.2 14.4 0.7X
Array: Int unsafe:false 3012 / 3032 5.2 191.5 0.1X
Array: Long unsafe:false 4463 / 4515 3.5 283.8 0.0X
Array: Float unsafe:false 2788 / 2868 5.6 177.2 0.1X
Array: Double unsafe:false 3558 / 3752 4.4 226.2 0.0X
Map of string->Double unsafe:false 2806 / 2933 5.6 178.4 0.1X
```
Author: Sandeep Singh <sandeep@techaddict.me>
Author: Sandeep Singh <sandeep@origamilogic.com>
Closes #12913 from techaddict/SPARK-928.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configuration.md | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index a4a99d6fa4..b07867d99a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -800,6 +800,14 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.kryo.unsafe</code></td> + <td>false</td> + <td> + Whether to use unsafe based Kryo serializer. Can be + substantially faster by using Unsafe Based IO. + </td> +</tr> +<tr> <td><code>spark.kryoserializer.buffer.max</code></td> <td>64m</td> <td> |