diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2016-02-19 15:57:23 -0800 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2016-02-19 15:57:23 -0800 |
| commit | 983fa2d62029e7334fb661cb65c8cadaa4b86d1c (patch) | |
| tree | 821694b53512200930a84753a0bafb982e2e85e8 /python/pyspark/sql | |
| parent | 091f6a7830bbee01fa580fbb0336b9f4fcac0dfa (diff) | |
| download | spark-983fa2d62029e7334fb661cb65c8cadaa4b86d1c.tar.gz spark-983fa2d62029e7334fb661cb65c8cadaa4b86d1c.tar.bz2 spark-983fa2d62029e7334fb661cb65c8cadaa4b86d1c.zip | |
[SPARK-13407] Guard against garbage-collected accumulators in TaskMetrics.fromAccumulatorUpdates
`TaskMetrics.fromAccumulatorUpdates()` can fail if accumulators have been garbage-collected on the driver. To guard against this, this patch introduces `ListenerTaskMetrics`, a subclass of `TaskMetrics` which is used only in `TaskMetrics.fromAccumulatorUpdates()` and which eliminates the need to access the original accumulators on the driver.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #11276 from JoshRosen/accum-updates-fix.
Diffstat (limited to 'python/pyspark/sql')
0 files changed, 0 insertions, 0 deletions