diff options
| author | Shixiong Zhu <shixiong@databricks.com> | 2016-06-09 18:45:19 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2016-06-09 18:45:19 -0700 |
| commit | 4d9d9cc5853c467acdb67915117127915a98d8f8 (patch) | |
| tree | 41760d6632935f393785a730591813b9730cf06c | |
| parent | b914e1930fd5c5f2808f92d4958ec6fbeddf2e30 (diff) | |
| download | spark-4d9d9cc5853c467acdb67915117127915a98d8f8.tar.gz spark-4d9d9cc5853c467acdb67915117127915a98d8f8.tar.bz2 spark-4d9d9cc5853c467acdb67915117127915a98d8f8.zip | |
[SPARK-15853][SQL] HDFSMetadataLog.get should close the input stream
## What changes were proposed in this pull request?
This PR closes the input stream created in `HDFSMetadataLog.get`
## How was this patch tested?
Jenkins unit tests.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #13583 from zsxwing/leak.
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala | 8 |
1 files changed, 6 insertions, 2 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala index fca3d51535..069e41b6ce 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala @@ -175,8 +175,12 @@ class HDFSMetadataLog[T: ClassTag](sparkSession: SparkSession, path: String) val batchMetadataFile = batchIdToPath(batchId) if (fileManager.exists(batchMetadataFile)) { val input = fileManager.open(batchMetadataFile) - val bytes = IOUtils.toByteArray(input) - Some(deserialize(bytes)) + try { + val bytes = IOUtils.toByteArray(input) + Some(deserialize(bytes)) + } finally { + input.close() + } } else { logDebug(s"Unable to find batch $batchMetadataFile") None |