diff options
| author | Tathagata Das <tathagata.das1565@gmail.com> | 2016-05-10 16:43:32 -0700 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2016-05-10 16:43:32 -0700 |
| commit | d9ca9fd3e582f9d29f8887c095637c93a8b93651 (patch) | |
| tree | 94ef5e8c0111fd438662ca706df85a95e92b5917 /python | |
| parent | 86475520f88f90c9d3b71516f65ccc0e9d244863 (diff) | |
| download | spark-d9ca9fd3e582f9d29f8887c095637c93a8b93651.tar.gz spark-d9ca9fd3e582f9d29f8887c095637c93a8b93651.tar.bz2 spark-d9ca9fd3e582f9d29f8887c095637c93a8b93651.zip | |
[SPARK-14837][SQL][STREAMING] Added support in file stream source for reading new files added to subdirs
## What changes were proposed in this pull request?
Currently, file stream source can only find new files if they appear in the directory given to the source, but not if they appear in subdirs. This PR add support for providing glob patterns when creating file stream source so that it can find new files in nested directories based on the glob pattern.
## How was this patch tested?
Unit test that tests when new files are discovered with globs and partitioned directories.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #12616 from tdas/SPARK-14837.
Diffstat (limited to 'python')
0 files changed, 0 insertions, 0 deletions