diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2016-09-15 11:22:58 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2016-09-15 11:22:58 -0700 |
| commit | 5b8f7377d54f83b93ef2bfc2a01ca65fae6d3032 (patch) | |
| tree | a1d7981f846b586fc56dff7e189012c4d06d3530 /sql/core | |
| parent | 0ad8eeb4d365c2fff5715ec22fbcf4c69c3340fd (diff) | |
| download | spark-5b8f7377d54f83b93ef2bfc2a01ca65fae6d3032.tar.gz spark-5b8f7377d54f83b93ef2bfc2a01ca65fae6d3032.tar.bz2 spark-5b8f7377d54f83b93ef2bfc2a01ca65fae6d3032.zip | |
[SPARK-17547] Ensure temp shuffle data file is cleaned up after error
SPARK-8029 (#9610) modified shuffle writers to first stage their data to a temporary file in the same directory as the final destination file and then to atomically rename this temporary file at the end of the write job. However, this change introduced the potential for the temporary output file to be leaked if an exception occurs during the write because the shuffle writers' existing error cleanup code doesn't handle deletion of the temp file.
This patch avoids this potential cause of disk-space leaks by adding `finally` blocks to ensure that temp files are always deleted if they haven't been renamed.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #15104 from JoshRosen/cleanup-tmp-data-file-in-shuffle-writer.
Diffstat (limited to 'sql/core')
0 files changed, 0 insertions, 0 deletions