From dd95abada78b4d0aec97dacda50fdfd74464b073 Mon Sep 17 00:00:00 2001 From: Reynold Xin Date: Tue, 15 Jul 2014 01:46:57 -0700 Subject: [SPARK-2399] Add support for LZ4 compression. Based on Greg Bowyer's patch from JIRA https://issues.apache.org/jira/browse/SPARK-2399 Author: Reynold Xin Closes #1416 from rxin/lz4 and squashes the following commits: 6c8fefe [Reynold Xin] Fixed typo. 8a14d38 [Reynold Xin] [SPARK-2399] Add support for LZ4 compression. --- docs/configuration.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) (limited to 'docs/configuration.md') diff --git a/docs/configuration.md b/docs/configuration.md index 07aa4c0354..19fd980e60 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -350,7 +350,15 @@ Apart from these, the following properties are also available, and may be useful 32768 Block size (in bytes) used in Snappy compression, in the case when Snappy compression codec - is used. + is used. Lowering this block size will also lower shuffle memory usage when Snappy is used. + + + + spark.io.compression.lz4.block.size + 32768 + + Block size (in bytes) used in LZ4 compression, in the case when LZ4 compression codec + is used. Lowering this block size will also lower shuffle memory usage when LZ4 is used. -- cgit v1.2.3