diff options
| author | Patrick Wendell <pwendell@gmail.com> | 2014-01-20 19:00:48 -0800 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-01-20 19:00:48 -0800 |
| commit | c324ac10ee208c53dabd54e5c0e1885aea456811 (patch) | |
| tree | 485f5f8facbae362b723f5f00270e4fad66cdd8d /docs/configuration.md | |
| parent | 1b299142a8d5feb70677dce993127de466266ff6 (diff) | |
| download | spark-c324ac10ee208c53dabd54e5c0e1885aea456811.tar.gz spark-c324ac10ee208c53dabd54e5c0e1885aea456811.tar.bz2 spark-c324ac10ee208c53dabd54e5c0e1885aea456811.zip | |
Force use of LZF when spilling data
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 4c2e9cc479..be548e372d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -158,7 +158,9 @@ Apart from these, the following properties are also available, and may be useful <td>spark.shuffle.spill.compress</td> <td>true</td> <td> - Whether to compress data spilled during shuffles. + Whether to compress data spilled during shuffles. If enabled, spill compression + always uses the `org.apache.spark.io.LZFCompressionCodec` codec, + regardless of the value of `spark.io.compression.codec`. </td> </tr> <tr> |