diff options
author | Yu Peng <loneknightpy@gmail.com> | 2016-10-18 13:23:31 -0700 |
---|---|---|
committer | Tathagata Das <tathagata.das1565@gmail.com> | 2016-10-18 13:23:31 -0700 |
commit | 231f39e3f6641953a90bc4c40444ede63f363b23 (patch) | |
tree | 679bf48c57a3e00196e9f610d1bc0bf386bcd97f /docs | |
parent | 37686539f546ac7a3657dbfc59b7ac982b4b9bce (diff) | |
download | spark-231f39e3f6641953a90bc4c40444ede63f363b23.tar.gz spark-231f39e3f6641953a90bc4c40444ede63f363b23.tar.bz2 spark-231f39e3f6641953a90bc4c40444ede63f363b23.zip |
[SPARK-17711] Compress rolled executor log
## What changes were proposed in this pull request?
This PR adds support for executor log compression.
## How was this patch tested?
Unit tests
cc: yhuai tdas mengxr
Author: Yu Peng <loneknightpy@gmail.com>
Closes #15285 from loneknightpy/compress-executor-log.
Diffstat (limited to 'docs')
-rw-r--r-- | docs/configuration.md | 8 | ||||
-rw-r--r-- | docs/spark-standalone.md | 9 |
2 files changed, 17 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 373e22d71a..a4a99d6fa4 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -294,6 +294,14 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.executor.logs.rolling.enableCompression</code></td> + <td>false</td> + <td> + Enable executor log compression. If it is enabled, the rolled executor logs will be compressed. + Disabled by default. + </td> +</tr> +<tr> <td><code>spark.executor.logs.rolling.maxSize</code></td> <td>(none)</td> <td> diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index 7b82b957d5..1c0b60f7b9 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -250,6 +250,15 @@ SPARK_WORKER_OPTS supports the following system properties: especially if you run jobs very frequently. </td> </tr> +<tr> + <td><code>spark.worker.ui.compressedLogFileLengthCacheSize</code></td> + <td>100</td> + <td> + For compressed log files, the uncompressed file can only be computed by uncompressing the files. + Spark caches the uncompressed file size of compressed log files. This property controls the cache + size. + </td> +</tr> </table> # Connecting an Application to the Cluster |