diff options
| author | Shixiong Zhu <shixiong@databricks.com> | 2017-03-07 20:32:51 -0800 |
|---|---|---|
| committer | Shixiong Zhu <shixiong@databricks.com> | 2017-03-07 20:32:51 -0800 |
| commit | ca849ac4e8fc520a4a12949b62b9730c5dfa097d (patch) | |
| tree | 0fdf56fbbd9bc7b09d56dd023bd0f1bf68269ab5 /mllib | |
| parent | b9783a92f7ba0c3b22d7dceae7a3185de17dedcc (diff) | |
| download | spark-ca849ac4e8fc520a4a12949b62b9730c5dfa097d.tar.gz spark-ca849ac4e8fc520a4a12949b62b9730c5dfa097d.tar.bz2 spark-ca849ac4e8fc520a4a12949b62b9730c5dfa097d.zip | |
[SPARK-19841][SS] watermarkPredicate should filter based on keys
## What changes were proposed in this pull request?
`StreamingDeduplicateExec.watermarkPredicate` should filter based on keys. Otherwise, it may generate a wrong answer if the watermark column in `keyExpression` has a different position in the row.
`StateStoreSaveExec` has the same codes but its parent can makes sure the watermark column positions in `keyExpression` and `row` are the same.
## How was this patch tested?
The added test.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #17183 from zsxwing/SPARK-19841.
Diffstat (limited to 'mllib')
0 files changed, 0 insertions, 0 deletions