diff options
author | Reynold Xin <rxin@databricks.com> | 2016-04-14 10:54:57 -0700 |
---|---|---|
committer | Andrew Or <andrew@databricks.com> | 2016-04-14 10:54:57 -0700 |
commit | dac40b68dc52d5ab855dfde63f0872064aa3d999 (patch) | |
tree | c9192854e5f40d52d3a2ce1096da370b79888560 /sql | |
parent | 9fa43a33b91c3a9b6be39bf3e00febf61a4b5b59 (diff) | |
download | spark-dac40b68dc52d5ab855dfde63f0872064aa3d999.tar.gz spark-dac40b68dc52d5ab855dfde63f0872064aa3d999.tar.bz2 spark-dac40b68dc52d5ab855dfde63f0872064aa3d999.zip |
[SPARK-14619] Track internal accumulators (metrics) by stage attempt
## What changes were proposed in this pull request?
When there are multiple attempts for a stage, we currently only reset internal accumulator values if all the tasks are resubmitted. It would make more sense to reset the accumulator values for each stage attempt. This will allow us to eventually get rid of the internal flag in the Accumulator class. This is part of my bigger effort to simplify accumulators and task metrics.
## How was this patch tested?
Covered by existing tests.
Author: Reynold Xin <rxin@databricks.com>
Closes #12378 from rxin/SPARK-14619.
Diffstat (limited to 'sql')
-rw-r--r-- | sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeRowSerializerSuite.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeRowSerializerSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeRowSerializerSuite.scala index 7db1f9654b..01687877ee 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeRowSerializerSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeRowSerializerSuite.scala @@ -114,7 +114,7 @@ class UnsafeRowSerializerSuite extends SparkFunSuite with LocalSparkContext { } val taskMemoryManager = new TaskMemoryManager(sc.env.memoryManager, 0) val taskContext = new TaskContextImpl( - 0, 0, 0, 0, taskMemoryManager, new Properties, null, InternalAccumulator.create(sc)) + 0, 0, 0, 0, taskMemoryManager, new Properties, null, InternalAccumulator.createAll(sc)) val sorter = new ExternalSorter[Int, UnsafeRow, UnsafeRow]( taskContext, |