diff options
| author | Shixiong Zhu <shixiong@databricks.com> | 2016-11-18 16:34:38 -0800 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2016-11-18 16:34:38 -0800 |
| commit | 2a40de408b5eb47edba92f9fe92a42ed1e78bf98 (patch) | |
| tree | 7ba9f245b65b29d8fff7e298c36582660fcb16f6 /python/pyspark/rdd.py | |
| parent | 6f7ff75091154fed7649ea6d79e887aad9fbde6a (diff) | |
| download | spark-2a40de408b5eb47edba92f9fe92a42ed1e78bf98.tar.gz spark-2a40de408b5eb47edba92f9fe92a42ed1e78bf98.tar.bz2 spark-2a40de408b5eb47edba92f9fe92a42ed1e78bf98.zip | |
[SPARK-18497][SS] Make ForeachSink support watermark
## What changes were proposed in this pull request?
The issue in ForeachSink is the new created DataSet still uses the old QueryExecution. When `foreachPartition` is called, `QueryExecution.toString` will be called and then fail because it doesn't know how to plan EventTimeWatermark.
This PR just replaces the QueryExecution with IncrementalExecution to fix the issue.
## How was this patch tested?
`test("foreach with watermark")`.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #15934 from zsxwing/SPARK-18497.
Diffstat (limited to 'python/pyspark/rdd.py')
0 files changed, 0 insertions, 0 deletions