diff options
author | Koert Kuipers <koert@tresata.com> | 2014-07-30 00:18:59 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-07-30 00:26:14 -0700 |
commit | 7c5fc28af42daaa6725af083d78c2372f3d0a338 (patch) | |
tree | b11558abc44907a58902da9a6b32c4ac40e16a39 /docs | |
parent | 7003c163dbb46bb7313aab130a33486a356435a8 (diff) | |
download | spark-7c5fc28af42daaa6725af083d78c2372f3d0a338.tar.gz spark-7c5fc28af42daaa6725af083d78c2372f3d0a338.tar.bz2 spark-7c5fc28af42daaa6725af083d78c2372f3d0a338.zip |
SPARK-2543: Allow user to set maximum Kryo buffer size
Author: Koert Kuipers <koert@tresata.com>
Closes #735 from koertkuipers/feat-kryo-max-buffersize and squashes the following commits:
15f6d81 [Koert Kuipers] change default for spark.kryoserializer.buffer.max.mb to 64mb and add some documentation
1bcc22c [Koert Kuipers] Merge branch 'master' into feat-kryo-max-buffersize
0c9f8eb [Koert Kuipers] make default for kryo max buffer size 16MB
143ec4d [Koert Kuipers] test resizable buffer in kryo Output
0732445 [Koert Kuipers] support setting maxCapacity to something different than capacity in kryo Output
Diffstat (limited to 'docs')
-rw-r--r-- | docs/configuration.md | 16 |
1 files changed, 12 insertions, 4 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 2e6c85cc2b..ea69057b5b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -414,10 +414,18 @@ Apart from these, the following properties are also available, and may be useful <td><code>spark.kryoserializer.buffer.mb</code></td> <td>2</td> <td> - Maximum object size to allow within Kryo (the library needs to create a buffer at least as - large as the largest single object you'll serialize). Increase this if you get a "buffer limit - exceeded" exception inside Kryo. Note that there will be one buffer <i>per core</i> on each - worker. + Initial size of Kryo's serialization buffer, in megabytes. Note that there will be one buffer + <i>per core</i> on each worker. This buffer will grow up to + <code>spark.kryoserializer.buffer.max.mb</code> if needed. + </td> +</tr> +<tr> + <td><code>spark.kryoserializer.buffer.max.mb</code></td> + <td>64</td> + <td> + Maximum allowable size of Kryo serialization buffer, in megabytes. This must be larger than any + object you attempt to serialize. Increase this if you get a "buffer limit exceeded" exception + inside Kryo. </td> </tr> </table> |