diff options
| author | Ryan Blue <blue@apache.org> | 2016-07-08 12:37:26 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-07-08 12:37:26 -0700 |
| commit | 67e085ef6dd62774095f3187844c091db1a6a72c (patch) | |
| tree | f403345e93f19a58bed6eb35151a9e9b90d6c2e0 /python/pyspark | |
| parent | 38cf8f2a50068f80350740ac28e31c8accd20634 (diff) | |
| download | spark-67e085ef6dd62774095f3187844c091db1a6a72c.tar.gz spark-67e085ef6dd62774095f3187844c091db1a6a72c.tar.bz2 spark-67e085ef6dd62774095f3187844c091db1a6a72c.zip | |
[SPARK-16420] Ensure compression streams are closed.
## What changes were proposed in this pull request?
This uses the try/finally pattern to ensure streams are closed after use. `UnsafeShuffleWriter` wasn't closing compression streams, causing them to leak resources until garbage collected. This was causing a problem with codecs that use off-heap memory.
## How was this patch tested?
Current tests are sufficient. This should not change behavior.
Author: Ryan Blue <blue@apache.org>
Closes #14093 from rdblue/SPARK-16420-unsafe-shuffle-writer-leak.
Diffstat (limited to 'python/pyspark')
0 files changed, 0 insertions, 0 deletions