diff options
| author | Parth Brahmbhatt <pbrahmbhatt@netflix.com> | 2016-04-21 06:58:00 -0500 |
|---|---|---|
| committer | Tom Graves <tgraves@yahoo-inc.com> | 2016-04-21 06:58:00 -0500 |
| commit | 6fdd0e32a6c3fdce1f3f7e1f8d252af05c419f7b (patch) | |
| tree | 82efefcf3a1f2b0a998a829537279cfe79bbd9f8 /docs/monitoring.md | |
| parent | 4ac6e75cd6b2ab90dbf3bd47e8a6b65059e0f2f9 (diff) | |
| download | spark-6fdd0e32a6c3fdce1f3f7e1f8d252af05c419f7b.tar.gz spark-6fdd0e32a6c3fdce1f3f7e1f8d252af05c419f7b.tar.bz2 spark-6fdd0e32a6c3fdce1f3f7e1f8d252af05c419f7b.zip | |
[SPARK-13988][CORE] Make replaying event logs multi threaded in Histo…ry server to ensure a single large log does not block other logs from being rendered.
## What changes were proposed in this pull request?
The patch makes event log processing multi threaded.
## How was this patch tested?
Existing tests pass, there is no new tests needed to test the functionality as this is a perf improvement. I tested the patch locally by generating one big event log (big1), one small event log(small1) and again a big event log(big2). Without this patch UI does not render any app for almost 30 seconds and then big2 and small1 appears. another 30 second delay and finally big1 also shows up in UI. With this change small1 shows up immediately and big1 and big2 comes up in 30 seconds. Locally it also displays them in the correct order in the UI.
Author: Parth Brahmbhatt <pbrahmbhatt@netflix.com>
Closes #11800 from Parth-Brahmbhatt/SPARK-13988.
Diffstat (limited to 'docs/monitoring.md')
| -rw-r--r-- | docs/monitoring.md | 7 |
1 files changed, 7 insertions, 0 deletions
diff --git a/docs/monitoring.md b/docs/monitoring.md index 32d2e02e93..9dcb0708d7 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -189,6 +189,13 @@ The history server can be configured as follows: Job history files older than this will be deleted when the filesystem history cleaner runs. </td> </tr> + <tr> + <td>spark.history.fs.numReplayThreads</td> + <td>25% of available cores</td> + <td> + Number of threads that will be used by history server to process event logs. + </td> + </tr> </table> Note that in all of these UIs, the tables are sortable by clicking their headers, |