diff options
author | Kostas Sakellis <kostas@cloudera.com> | 2015-02-06 14:31:20 -0800 |
---|---|---|
committer | Patrick Wendell <patrick@databricks.com> | 2015-02-06 14:31:20 -0800 |
commit | dcd1e42d6b6ac08d2c0736bf61a15f515a1f222b (patch) | |
tree | 13b498d7cc6fff743fce1719aef50d716d7677f2 /README.md | |
parent | 57961567ef104efb3174e67d762c5d9d6263b800 (diff) | |
download | spark-dcd1e42d6b6ac08d2c0736bf61a15f515a1f222b.tar.gz spark-dcd1e42d6b6ac08d2c0736bf61a15f515a1f222b.tar.bz2 spark-dcd1e42d6b6ac08d2c0736bf61a15f515a1f222b.zip |
[SPARK-4874] [CORE] Collect record count metrics
Collects record counts for both Input/Output and Shuffle Metrics. For the input/output metrics, it just appends the counter every time the iterators get accessed.
For shuffle on the write side, we count the metrics post aggregation (after a map side combine) and on the read side we count the metrics pre aggregation. This allows both the bytes read/written metrics and the records read/written to line up.
For backwards compatibility, if we deserialize an older event that doesn't have record metrics, we set the metric to -1.
Author: Kostas Sakellis <kostas@cloudera.com>
Closes #4067 from ksakellis/kostas-spark-4874 and squashes the following commits:
bd919be [Kostas Sakellis] Changed 'Records Read' in shuffleReadMetrics json output to 'Total Records Read'
dad4d57 [Kostas Sakellis] Add a comment and check to BlockObjectWriter so that it cannot be reopend.
6f236a1 [Kostas Sakellis] Renamed _recordsWritten in ShuffleWriteMetrics to be more consistent
70620a0 [Kostas Sakellis] CR Feedback
17faa3a [Kostas Sakellis] Removed AtomicLong in favour of using Long
b6f9923 [Kostas Sakellis] Merge AfterNextInterceptingIterator with InterruptableIterator to save a function call
46c8186 [Kostas Sakellis] Combined Bytes and # records into one column
57551c1 [Kostas Sakellis] Conforms to SPARK-3288
6cdb44e [Kostas Sakellis] Removed the generic InterceptingIterator and repalced it with specific implementation
1aa273c [Kostas Sakellis] CR Feedback
1bb78b1 [Kostas Sakellis] [SPARK-4874] [CORE] Collect record count metrics
Diffstat (limited to 'README.md')
0 files changed, 0 insertions, 0 deletions