diff options
| author | Brian Cho <bcho@fb.com> | 2016-10-07 11:37:18 -0400 |
|---|---|---|
| committer | Andrew Or <andrewor14@gmail.com> | 2016-10-07 11:37:18 -0400 |
| commit | e56614cba99bfdf5fa8a6c617fdd56eca2b34694 (patch) | |
| tree | 772c30f2a669a1b90ff5ded06b7e2d34322201ff | |
| parent | 2b01d3c701c58f07fa42afd570523dd161384882 (diff) | |
| download | spark-e56614cba99bfdf5fa8a6c617fdd56eca2b34694.tar.gz spark-e56614cba99bfdf5fa8a6c617fdd56eca2b34694.tar.bz2 spark-e56614cba99bfdf5fa8a6c617fdd56eca2b34694.zip | |
[SPARK-16827] Stop reporting spill metrics as shuffle metrics
## What changes were proposed in this pull request?
Fix a bug where spill metrics were being reported as shuffle metrics. Eventually these spill metrics should be reported (SPARK-3577), but separate from shuffle metrics. The fix itself basically reverts the line to what it was in 1.6.
## How was this patch tested?
Tested on a job that was reporting shuffle writes even for the final stage, when no shuffle writes should take place. After the change the job no longer shows these writes.
Before:
After:
<img width="1052" alt="screen shot 2016-10-03 at 11 44 44 pm" src="https://cloud.githubusercontent.com/assets/1514239/19085903/e173a860-8a20-11e6-85e3-d47f9835f494.png">
Author: Brian Cho <bcho@fb.com>
Closes #15347 from dafrista/shuffle-metrics.
| -rw-r--r-- | core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java index 428ff72e71..7835017910 100644 --- a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java +++ b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java @@ -145,7 +145,9 @@ public final class UnsafeExternalSorter extends MemoryConsumer { // Use getSizeAsKb (not bytes) to maintain backwards compatibility for units // this.fileBufferSizeBytes = (int) conf.getSizeAsKb("spark.shuffle.file.buffer", "32k") * 1024; this.fileBufferSizeBytes = 32 * 1024; - this.writeMetrics = taskContext.taskMetrics().shuffleWriteMetrics(); + // The spill metrics are stored in a new ShuffleWriteMetrics, and then discarded (this fixes SPARK-16827). + // TODO: Instead, separate spill metrics should be stored and reported (tracked in SPARK-3577). + this.writeMetrics = new ShuffleWriteMetrics(); if (existingInMemorySorter == null) { this.inMemSorter = new UnsafeInMemorySorter( |