diff options
| author | Tathagata Das <tathagata.das1565@gmail.com> | 2014-08-27 00:13:38 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-08-27 00:13:38 -0700 |
| commit | 3e2864e40472b32e6a7eec5ba3bc83562d2a1a62 (patch) | |
| tree | ae54f24c9974d1343bee71c8148294a2c3449c9e /sql/catalyst/src/main/scala/org/apache | |
| parent | 9d65f2712c250a561c9c1f6259aa12e861ed239d (diff) | |
| download | spark-3e2864e40472b32e6a7eec5ba3bc83562d2a1a62.tar.gz spark-3e2864e40472b32e6a7eec5ba3bc83562d2a1a62.tar.bz2 spark-3e2864e40472b32e6a7eec5ba3bc83562d2a1a62.zip | |
[SPARK-3139] Made ContextCleaner to not block on shuffles
As a workaround for SPARK-3015, the ContextCleaner was made "blocking", that is, it cleaned items one-by-one. But shuffles can take a long time to be deleted. Given that the RC for 1.1 is imminent, this PR makes a narrow change in the context cleaner - not wait for shuffle cleanups to complete. Also it changes the error messages on failure to delete to be milder warnings, as exceptions in the delete code path for one item does not really stop the actual functioning of the system.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #2143 from tdas/cleaner-shuffle-fix and squashes the following commits:
9c84202 [Tathagata Das] Restoring default blocking behavior in ContextCleanerSuite, and added docs to identify that spark.cleaner.referenceTracking.blocking does not control shuffle.
2181329 [Tathagata Das] Mark shuffle cleanup as non-blocking.
e337cc2 [Tathagata Das] Changed semantics based on PR comments.
387b578 [Tathagata Das] Made ContextCleaner to not block on shuffles
Diffstat (limited to 'sql/catalyst/src/main/scala/org/apache')
0 files changed, 0 insertions, 0 deletions