diff options
author | Reynold Xin <rxin@apache.org> | 2014-08-05 16:24:50 -0700 |
---|---|---|
committer | Reynold Xin <rxin@apache.org> | 2014-08-05 16:25:05 -0700 |
commit | 0172277d2aae3913b7ee5a71906427f905eda0b7 (patch) | |
tree | 8c0226b6eb12d4e480a240686f33e9b0d88d3361 /docs/configuration.md | |
parent | 5081b0da06232f2221c1e9d600f95a66f06ea357 (diff) | |
download | spark-0172277d2aae3913b7ee5a71906427f905eda0b7.tar.gz spark-0172277d2aae3913b7ee5a71906427f905eda0b7.tar.bz2 spark-0172277d2aae3913b7ee5a71906427f905eda0b7.zip |
[SPARK-2503] Lower shuffle output buffer (spark.shuffle.file.buffer.kb) to 32KB.
This can substantially reduce memory usage during shuffle.
Author: Reynold Xin <rxin@apache.org>
Closes #1781 from rxin/SPARK-2503-spark.shuffle.file.buffer.kb and squashes the following commits:
104b8d8 [Reynold Xin] [SPARK-2503] Lower shuffle output buffer (spark.shuffle.file.buffer.kb) to 32KB.
(cherry picked from commit acff9a7f13b98f10a08aea1d11cfa685c3419367)
Signed-off-by: Reynold Xin <rxin@apache.org>
Diffstat (limited to 'docs/configuration.md')
-rw-r--r-- | docs/configuration.md | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 5e7556c08e..7cd7f4124d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -266,7 +266,7 @@ Apart from these, the following properties are also available, and may be useful </tr> <tr> <td><code>spark.shuffle.file.buffer.kb</code></td> - <td>100</td> + <td>32</td> <td> Size of the in-memory buffer for each shuffle file output stream, in kilobytes. These buffers reduce the number of disk seeks and system calls made in creating intermediate shuffle files. |