aboutsummaryrefslogtreecommitdiff
path: root/docs/configuration.md
diff options
context:
space:
mode:
author郭小龙 10207633 <guo.xiaolong1@zte.com.cn>2017-04-01 11:48:58 +0100
committerSean Owen <sowen@cloudera.com>2017-04-01 11:48:58 +0100
commitcf5963c961e7eba37bdd58658ed4dfff66ce3c72 (patch)
tree66c84c4490a57aeddc694d212aff5efcd3bb77e0 /docs/configuration.md
parent567a50acfb0ae26bd430c290348886d494963696 (diff)
downloadspark-cf5963c961e7eba37bdd58658ed4dfff66ce3c72.tar.gz
spark-cf5963c961e7eba37bdd58658ed4dfff66ce3c72.tar.bz2
spark-cf5963c961e7eba37bdd58658ed4dfff66ce3c72.zip
[SPARK-20177] Document about compression way has some little detail ch…
…anges. ## What changes were proposed in this pull request? Document compression way little detail changes. 1.spark.eventLog.compress add 'Compression will use spark.io.compression.codec.' 2.spark.broadcast.compress add 'Compression will use spark.io.compression.codec.' 3,spark.rdd.compress add 'Compression will use spark.io.compression.codec.' 4.spark.io.compression.codec add 'event log describe'. eg Through the documents, I don't know what is compression mode about 'event log'. ## How was this patch tested? manual tests Please review http://spark.apache.org/contributing.html before opening a pull request. Author: 郭小龙 10207633 <guo.xiaolong1@zte.com.cn> Closes #17498 from guoxiaolongzte/SPARK-20177.
Diffstat (limited to 'docs/configuration.md')
-rw-r--r--docs/configuration.md7
1 files changed, 5 insertions, 2 deletions
diff --git a/docs/configuration.md b/docs/configuration.md
index a975392540..2687f542b8 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -639,6 +639,7 @@ Apart from these, the following properties are also available, and may be useful
<td>false</td>
<td>
Whether to compress logged events, if <code>spark.eventLog.enabled</code> is true.
+ Compression will use <code>spark.io.compression.codec</code>.
</td>
</tr>
<tr>
@@ -773,14 +774,15 @@ Apart from these, the following properties are also available, and may be useful
<td>true</td>
<td>
Whether to compress broadcast variables before sending them. Generally a good idea.
+ Compression will use <code>spark.io.compression.codec</code>.
</td>
</tr>
<tr>
<td><code>spark.io.compression.codec</code></td>
<td>lz4</td>
<td>
- The codec used to compress internal data such as RDD partitions, broadcast variables and
- shuffle outputs. By default, Spark provides three codecs: <code>lz4</code>, <code>lzf</code>,
+ The codec used to compress internal data such as RDD partitions, event log, broadcast variables
+ and shuffle outputs. By default, Spark provides three codecs: <code>lz4</code>, <code>lzf</code>,
and <code>snappy</code>. You can also use fully qualified class names to specify the codec,
e.g.
<code>org.apache.spark.io.LZ4CompressionCodec</code>,
@@ -881,6 +883,7 @@ Apart from these, the following properties are also available, and may be useful
<code>StorageLevel.MEMORY_ONLY_SER</code> in Java
and Scala or <code>StorageLevel.MEMORY_ONLY</code> in Python).
Can save substantial space at the cost of some extra CPU time.
+ Compression will use <code>spark.io.compression.codec</code>.
</td>
</tr>
<tr>