diff options
| author | jerryshao <sshao@hortonworks.com> | 2017-04-06 13:23:54 -0500 |
|---|---|---|
| committer | Imran Rashid <irashid@cloudera.com> | 2017-04-06 13:23:54 -0500 |
| commit | a4491626ed8169f0162a0dfb78736c9b9e7fb434 (patch) | |
| tree | c18ac0eeba2cb1e95ef6a6a2dee2a9c16ec31174 /examples | |
| parent | 5a693b4138d4ce948e3bcdbe28d5c01d5deb8fa9 (diff) | |
| download | spark-a4491626ed8169f0162a0dfb78736c9b9e7fb434.tar.gz spark-a4491626ed8169f0162a0dfb78736c9b9e7fb434.tar.bz2 spark-a4491626ed8169f0162a0dfb78736c9b9e7fb434.zip | |
[SPARK-17019][CORE] Expose on-heap and off-heap memory usage in various places
## What changes were proposed in this pull request?
With [SPARK-13992](https://issues.apache.org/jira/browse/SPARK-13992), Spark supports persisting data into off-heap memory, but the usage of on-heap and off-heap memory is not exposed currently, it is not so convenient for user to monitor and profile, so here propose to expose off-heap memory as well as on-heap memory usage in various places:
1. Spark UI's executor page will display both on-heap and off-heap memory usage.
2. REST request returns both on-heap and off-heap memory.
3. Also this can be gotten from MetricsSystem.
4. Last this usage can be obtained programmatically from SparkListener.
Attach the UI changes:
Backward compatibility is also considered for event-log and REST API. Old event log can still be replayed with off-heap usage displayed as 0. For REST API, only adds the new fields, so JSON backward compatibility can still be kept.
## How was this patch tested?
Unit test added and manual verification.
Author: jerryshao <sshao@hortonworks.com>
Closes #14617 from jerryshao/SPARK-17019.
Diffstat (limited to 'examples')
0 files changed, 0 insertions, 0 deletions