diff options
| author | petermaxlee <petermaxlee@gmail.com> | 2016-09-19 22:19:51 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-09-19 22:19:51 -0700 |
| commit | be9d57fc9d8b10e4234c01c06ed43fd7dd12c07b (patch) | |
| tree | a5ca485bc454b64cbc31dc2dc35b36004c9eda19 /python | |
| parent | 26145a5af9a88053c0eaf280206ca2621c8919f6 (diff) | |
| download | spark-be9d57fc9d8b10e4234c01c06ed43fd7dd12c07b.tar.gz spark-be9d57fc9d8b10e4234c01c06ed43fd7dd12c07b.tar.bz2 spark-be9d57fc9d8b10e4234c01c06ed43fd7dd12c07b.zip | |
[SPARK-17513][SQL] Make StreamExecution garbage-collect its metadata
## What changes were proposed in this pull request?
This PR modifies StreamExecution such that it discards metadata for batches that have already been fully processed. I used the purge method that was added as part of SPARK-17235.
This is based on work by frreiss in #15067, but fixed the test case along with some typos.
## How was this patch tested?
A new test case in StreamingQuerySuite. The test case would fail without the changes in this pull request.
Author: petermaxlee <petermaxlee@gmail.com>
Author: frreiss <frreiss@us.ibm.com>
Closes #15126 from petermaxlee/SPARK-17513.
Diffstat (limited to 'python')
0 files changed, 0 insertions, 0 deletions