diff options
| author | Andrew Or <andrew@databricks.com> | 2015-05-05 09:37:04 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-05-05 09:37:04 -0700 |
| commit | 1fdabf8dcdb31391fec3952d312eb0ac59ece43b (patch) | |
| tree | d098993869ad6b053228bec7d1907c474b1fadd0 /streaming | |
| parent | d4cb38aeb7412a353c6cbca2a9b8f9729afbaba7 (diff) | |
| download | spark-1fdabf8dcdb31391fec3952d312eb0ac59ece43b.tar.gz spark-1fdabf8dcdb31391fec3952d312eb0ac59ece43b.tar.bz2 spark-1fdabf8dcdb31391fec3952d312eb0ac59ece43b.zip | |
[SPARK-7237] Many user provided closures are not actually cleaned
Note: ~140 lines are tests.
In a nutshell, we never cleaned closures the user provided through the following operations:
- sortBy
- keyBy
- mapPartitions
- mapPartitionsWithIndex
- aggregateByKey
- foldByKey
- foreachAsync
- one of the aliases for runJob
- runApproximateJob
For more details on a reproduction and why they were not cleaned, please see [SPARK-7237](https://issues.apache.org/jira/browse/SPARK-7237).
Author: Andrew Or <andrew@databricks.com>
Closes #5787 from andrewor14/clean-more and squashes the following commits:
2f1f476 [Andrew Or] Merge branch 'master' of github.com:apache/spark into clean-more
7265865 [Andrew Or] Merge branch 'master' of github.com:apache/spark into clean-more
df3caa3 [Andrew Or] Address comments
7a3cc80 [Andrew Or] Merge branch 'master' of github.com:apache/spark into clean-more
6498f44 [Andrew Or] Add missing test for groupBy
e83699e [Andrew Or] Clean one more
8ac3074 [Andrew Or] Prevent NPE in tests when CC is used outside of an app
9ac5f9b [Andrew Or] Clean closures that are not currently cleaned
19e33b4 [Andrew Or] Add tests for all public RDD APIs that take in closures
Diffstat (limited to 'streaming')
0 files changed, 0 insertions, 0 deletions