diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2015-02-28 22:51:01 -0800 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-02-28 22:51:01 -0800 |
| commit | 2df5f1f00661cd31b9fc37e80345a3f5f856c95f (patch) | |
| tree | 710eea3dc03ece9c3d5c912aacc4b2de86c54610 /mllib | |
| parent | 643300a6e27dac3822f9a3ced0ad5fb3b4f2ad75 (diff) | |
| download | spark-2df5f1f00661cd31b9fc37e80345a3f5f856c95f.tar.gz spark-2df5f1f00661cd31b9fc37e80345a3f5f856c95f.tar.bz2 spark-2df5f1f00661cd31b9fc37e80345a3f5f856c95f.zip | |
[SPARK-6075] Fix bug in that caused lost accumulator updates: do not store WeakReferences in localAccums map
This fixes a non-deterministic bug introduced in #4021 that could cause tasks' accumulator updates to be lost. The problem is that `localAccums` should not hold weak references: after the task finishes running there won't be any strong references to these local accumulators, so they can get garbage-collected before the executor reads the `localAccums` map. We don't need weak references here anyways, since this map is cleared at the end of each task.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #4835 from JoshRosen/SPARK-6075 and squashes the following commits:
4f4b5b2 [Josh Rosen] Remove defensive assertions that caused test failures in code unrelated to this change
120c7b0 [Josh Rosen] [SPARK-6075] Do not store WeakReferences in localAccums map
Diffstat (limited to 'mllib')
0 files changed, 0 insertions, 0 deletions