diff options
author | jerryshao <sshao@hortonworks.com> | 2017-04-06 13:23:54 -0500 |
---|---|---|
committer | Imran Rashid <irashid@cloudera.com> | 2017-04-06 13:23:54 -0500 |
commit | a4491626ed8169f0162a0dfb78736c9b9e7fb434 (patch) | |
tree | c18ac0eeba2cb1e95ef6a6a2dee2a9c16ec31174 /sql/hive/src/test | |
parent | 5a693b4138d4ce948e3bcdbe28d5c01d5deb8fa9 (diff) | |
download | spark-a4491626ed8169f0162a0dfb78736c9b9e7fb434.tar.gz spark-a4491626ed8169f0162a0dfb78736c9b9e7fb434.tar.bz2 spark-a4491626ed8169f0162a0dfb78736c9b9e7fb434.zip |
[SPARK-17019][CORE] Expose on-heap and off-heap memory usage in various places
## What changes were proposed in this pull request?
With [SPARK-13992](https://issues.apache.org/jira/browse/SPARK-13992), Spark supports persisting data into off-heap memory, but the usage of on-heap and off-heap memory is not exposed currently, it is not so convenient for user to monitor and profile, so here propose to expose off-heap memory as well as on-heap memory usage in various places:
1. Spark UI's executor page will display both on-heap and off-heap memory usage.
2. REST request returns both on-heap and off-heap memory.
3. Also this can be gotten from MetricsSystem.
4. Last this usage can be obtained programmatically from SparkListener.
Attach the UI changes:
![screen shot 2016-08-12 at 11 20 44 am](https://cloud.githubusercontent.com/assets/850797/17612032/6c2f4480-607f-11e6-82e8-a27fb8cbb4ae.png)
Backward compatibility is also considered for event-log and REST API. Old event log can still be replayed with off-heap usage displayed as 0. For REST API, only adds the new fields, so JSON backward compatibility can still be kept.
## How was this patch tested?
Unit test added and manual verification.
Author: jerryshao <sshao@hortonworks.com>
Closes #14617 from jerryshao/SPARK-17019.
Diffstat (limited to 'sql/hive/src/test')
0 files changed, 0 insertions, 0 deletions