diff options
| author | Alex Bozarth <ajbozart@us.ibm.com> | 2016-08-24 14:39:41 -0500 |
|---|---|---|
| committer | Tom Graves <tgraves@yahoo-inc.com> | 2016-08-24 14:39:41 -0500 |
| commit | 891ac2b914fb6f90a62c6fbc0a3960a89d1c1d92 (patch) | |
| tree | 19d1eea2303c7b29c310047d40e04a1b5f694cc7 /docs | |
| parent | 40b30fcf453169534cb53d01cd22236210b13005 (diff) | |
| download | spark-891ac2b914fb6f90a62c6fbc0a3960a89d1c1d92.tar.gz spark-891ac2b914fb6f90a62c6fbc0a3960a89d1c1d92.tar.bz2 spark-891ac2b914fb6f90a62c6fbc0a3960a89d1c1d92.zip | |
[SPARK-15083][WEB UI] History Server can OOM due to unlimited TaskUIData
## What changes were proposed in this pull request?
Based on #12990 by tankkyo
Since the History Server currently loads all application's data it can OOM if too many applications have a significant task count. `spark.ui.trimTasks` (default: false) can be set to true to trim tasks by `spark.ui.retainedTasks` (default: 10000)
(This is a "quick fix" to help those running into the problem until a update of how the history server loads app data can be done)
## How was this patch tested?
Manual testing and dev/run-tests
Author: Alex Bozarth <ajbozart@us.ibm.com>
Closes #14673 from ajbozarth/spark15083.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configuration.md | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 4bda464b98..2f80196105 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -627,6 +627,14 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.ui.retainedTasks</code></td> + <td>100000</td> + <td> + How many tasks the Spark UI and status APIs remember before garbage + collecting. + </td> +</tr> +<tr> <td><code>spark.worker.ui.retainedExecutors</code></td> <td>1000</td> <td> |