diff options
| author | Gabriele Nizzoli <mail@nizzoli.net> | 2014-07-08 14:23:38 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2014-07-08 14:23:38 -0700 |
| commit | e6f7bfcfbf6aff7a9f8cd8e0a2166d0bf62b0912 (patch) | |
| tree | 2887a43242dda775a8dd7513f3c9290643946b0a /streaming | |
| parent | 32516f866a32d51bfaa04685ae77ba216b4202d9 (diff) | |
| download | spark-e6f7bfcfbf6aff7a9f8cd8e0a2166d0bf62b0912.tar.gz spark-e6f7bfcfbf6aff7a9f8cd8e0a2166d0bf62b0912.tar.bz2 spark-e6f7bfcfbf6aff7a9f8cd8e0a2166d0bf62b0912.zip | |
[SPARK-2362] Fix for newFilesOnly logic in file DStream
The newFilesOnly logic should be inverted: the logic should be that if the flag newFilesOnly==true then only start reading files older than current time. As the code is now if newFilesOnly==true then it will start to read files that are older than 0L (that is: every file in the directory).
Author: Gabriele Nizzoli <mail@nizzoli.net>
Closes #1077 from gabrielenizzoli/master and squashes the following commits:
4f1d261 [Gabriele Nizzoli] Fix for newFilesOnly logic in file DStream
Diffstat (limited to 'streaming')
| -rw-r--r-- | streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala b/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala index e878285f6a..9eecbfaef3 100644 --- a/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala +++ b/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala @@ -45,7 +45,7 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: NewInputFormat[K,V] : Clas // Files with mod time earlier than this is ignored. This is updated every interval // such that in the current interval, files older than any file found in the // previous interval will be ignored. Obviously this time keeps moving forward. - private var ignoreTime = if (newFilesOnly) 0L else System.currentTimeMillis() + private var ignoreTime = if (newFilesOnly) System.currentTimeMillis() else 0L // Latest file mod time seen till any point of time @transient private var path_ : Path = null |