diff options
| author | Tathagata Das <tathagata.das1565@gmail.com> | 2014-11-24 13:50:20 -0800 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2014-11-24 13:50:20 -0800 |
| commit | cb0e9b0980f38befe88bf52aa037fe33262730f7 (patch) | |
| tree | 730da28c128641d8b9f82f0a8bb93e0c4c2d6f49 /python | |
| parent | 4a90276ab22d6989dffb2ee2d8118d9253365646 (diff) | |
| download | spark-cb0e9b0980f38befe88bf52aa037fe33262730f7.tar.gz spark-cb0e9b0980f38befe88bf52aa037fe33262730f7.tar.bz2 spark-cb0e9b0980f38befe88bf52aa037fe33262730f7.zip | |
[SPARK-4518][SPARK-4519][Streaming] Refactored file stream to prevent files from being processed multiple times
Because of a corner case, a file already selected for batch t can get considered again for batch t+2. This refactoring fixes it by remembering all the files selected in the last 1 minute, so that this corner case does not arise. Also uses spark context's hadoop configuration to access the file system API for listing directories.
pwendell Please take look. I still have not run long-running integration tests, so I cannot say for sure whether this has indeed solved the issue. You could do a first pass on this in the meantime.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #3419 from tdas/filestream-fix2 and squashes the following commits:
c19dd8a [Tathagata Das] Addressed PR comments.
513b608 [Tathagata Das] Updated docs.
d364faf [Tathagata Das] Added the current time condition back
5526222 [Tathagata Das] Removed unnecessary imports.
38bb736 [Tathagata Das] Fix long line.
203bbc7 [Tathagata Das] Un-ignore tests.
eaef4e1 [Tathagata Das] Fixed SPARK-4519
9dbd40a [Tathagata Das] Refactored FileInputDStream to remember last few batches.
Diffstat (limited to 'python')
0 files changed, 0 insertions, 0 deletions