diff options
author | 郭小龙 10207633 <guo.xiaolong1@zte.com.cn> | 2017-04-01 11:48:58 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2017-04-01 11:48:58 +0100 |
commit | cf5963c961e7eba37bdd58658ed4dfff66ce3c72 (patch) | |
tree | 66c84c4490a57aeddc694d212aff5efcd3bb77e0 /docs/configuration.md | |
parent | 567a50acfb0ae26bd430c290348886d494963696 (diff) | |
download | spark-cf5963c961e7eba37bdd58658ed4dfff66ce3c72.tar.gz spark-cf5963c961e7eba37bdd58658ed4dfff66ce3c72.tar.bz2 spark-cf5963c961e7eba37bdd58658ed4dfff66ce3c72.zip |
[SPARK-20177] Document about compression way has some little detail ch…
…anges.
## What changes were proposed in this pull request?
Document compression way little detail changes.
1.spark.eventLog.compress add 'Compression will use spark.io.compression.codec.'
2.spark.broadcast.compress add 'Compression will use spark.io.compression.codec.'
3,spark.rdd.compress add 'Compression will use spark.io.compression.codec.'
4.spark.io.compression.codec add 'event log describe'.
eg
Through the documents, I don't know what is compression mode about 'event log'.
## How was this patch tested?
manual tests
Please review http://spark.apache.org/contributing.html before opening a pull request.
Author: 郭小龙 10207633 <guo.xiaolong1@zte.com.cn>
Closes #17498 from guoxiaolongzte/SPARK-20177.
Diffstat (limited to 'docs/configuration.md')
-rw-r--r-- | docs/configuration.md | 7 |
1 files changed, 5 insertions, 2 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index a975392540..2687f542b8 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -639,6 +639,7 @@ Apart from these, the following properties are also available, and may be useful <td>false</td> <td> Whether to compress logged events, if <code>spark.eventLog.enabled</code> is true. + Compression will use <code>spark.io.compression.codec</code>. </td> </tr> <tr> @@ -773,14 +774,15 @@ Apart from these, the following properties are also available, and may be useful <td>true</td> <td> Whether to compress broadcast variables before sending them. Generally a good idea. + Compression will use <code>spark.io.compression.codec</code>. </td> </tr> <tr> <td><code>spark.io.compression.codec</code></td> <td>lz4</td> <td> - The codec used to compress internal data such as RDD partitions, broadcast variables and - shuffle outputs. By default, Spark provides three codecs: <code>lz4</code>, <code>lzf</code>, + The codec used to compress internal data such as RDD partitions, event log, broadcast variables + and shuffle outputs. By default, Spark provides three codecs: <code>lz4</code>, <code>lzf</code>, and <code>snappy</code>. You can also use fully qualified class names to specify the codec, e.g. <code>org.apache.spark.io.LZ4CompressionCodec</code>, @@ -881,6 +883,7 @@ Apart from these, the following properties are also available, and may be useful <code>StorageLevel.MEMORY_ONLY_SER</code> in Java and Scala or <code>StorageLevel.MEMORY_ONLY</code> in Python). Can save substantial space at the cost of some extra CPU time. + Compression will use <code>spark.io.compression.codec</code>. </td> </tr> <tr> |