diff options
| author | Andrew Or <andrewor14@gmail.com> | 2014-07-23 10:31:45 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-07-23 10:31:45 -0700 |
| commit | 25921110fcd5afe568bf0d25fccd232787af7911 (patch) | |
| tree | 3bbdd905d4ccae9639d155d9630549d7493b2e0b /scalastyle-config.xml | |
| parent | 4c7243e109c713bdfb87891748800109ffbaae07 (diff) | |
| download | spark-25921110fcd5afe568bf0d25fccd232787af7911.tar.gz spark-25921110fcd5afe568bf0d25fccd232787af7911.tar.bz2 spark-25921110fcd5afe568bf0d25fccd232787af7911.zip | |
[SPARK-2609] Log thread ID when spilling ExternalAppendOnlyMap
It's useful to know whether one thread is constantly spilling or multiple threads are spilling relatively infrequently. Right now everything looks a little jumbled and we can't tell which lines belong to the same thread. For instance:
```
06:14:37 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (194 times so far)
06:14:37 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (198 times so far)
06:14:37 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (198 times so far)
06:14:37 ExternalAppendOnlyMap: Spilling in-memory map of 10 MB to disk (197 times so far)
06:14:38 ExternalAppendOnlyMap: Spilling in-memory map of 9 MB to disk (45 times so far)
06:14:38 ExternalAppendOnlyMap: Spilling in-memory map of 23 MB to disk (198 times so far)
06:14:38 ExternalAppendOnlyMap: Spilling in-memory map of 38 MB to disk (25 times so far)
06:14:38 ExternalAppendOnlyMap: Spilling in-memory map of 161 MB to disk (25 times so far)
06:14:39 ExternalAppendOnlyMap: Spilling in-memory map of 0 MB to disk (199 times so far)
06:14:39 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (166 times so far)
06:14:39 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (199 times so far)
06:14:39 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (200 times so far)
```
Author: Andrew Or <andrewor14@gmail.com>
Closes #1517 from andrewor14/external-log and squashes the following commits:
90e48bb [Andrew Or] Log thread ID when spilling
Diffstat (limited to 'scalastyle-config.xml')
0 files changed, 0 insertions, 0 deletions