diff options
| author | Tathagata Das <tathagata.das1565@gmail.com> | 2016-04-18 11:09:33 -0700 |
|---|---|---|
| committer | Shixiong Zhu <shixiong@databricks.com> | 2016-04-18 11:09:33 -0700 |
| commit | 775cf17eaaae1a38efe47b282b1d6bbdb99bd759 (patch) | |
| tree | da02930091ff58ad899b8d13f06fc9647fce4b9a /python/pyspark/ml/classification.py | |
| parent | 432d1399cb6985893932088875b2f3be981c0b5f (diff) | |
| download | spark-775cf17eaaae1a38efe47b282b1d6bbdb99bd759.tar.gz spark-775cf17eaaae1a38efe47b282b1d6bbdb99bd759.tar.bz2 spark-775cf17eaaae1a38efe47b282b1d6bbdb99bd759.zip | |
[SPARK-14473][SQL] Define analysis rules to catch operations not supported in streaming
## What changes were proposed in this pull request?
There are many operations that are currently not supported in the streaming execution. For example:
- joining two streams
- unioning a stream and a batch source
- sorting
- window functions (not time windows)
- distinct aggregates
Furthermore, executing a query with a stream source as a batch query should also fail.
This patch add an additional step after analysis in the QueryExecution which will check that all the operations in the analyzed logical plan is supported or not.
## How was this patch tested?
unit tests.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #12246 from tdas/SPARK-14473.
Diffstat (limited to 'python/pyspark/ml/classification.py')
0 files changed, 0 insertions, 0 deletions