diff options
| author | Wenchen Fan <wenchen@databricks.com> | 2015-10-30 12:17:51 +0100 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2015-10-30 12:17:51 +0100 |
| commit | 14d08b99085d4e609aeae0cf54d4584e860eb552 (patch) | |
| tree | 1eed67ed7c0e7abaa9395122fd74262fb72a5768 /core | |
| parent | 59db9e9c382fab40aac0633f2c779bee8cf2025f (diff) | |
| download | spark-14d08b99085d4e609aeae0cf54d4584e860eb552.tar.gz spark-14d08b99085d4e609aeae0cf54d4584e860eb552.tar.bz2 spark-14d08b99085d4e609aeae0cf54d4584e860eb552.zip | |
[SPARK-11393] [SQL] CoGroupedIterator should respect the fact that GroupedIterator.hasNext is not idempotent
When we cogroup 2 `GroupedIterator`s in `CoGroupedIterator`, if the right side is smaller, we will consume right data and keep the left data unchanged. Then we call `hasNext` which will call `left.hasNext`. This will make `GroupedIterator` generate an extra group as the previous one has not been comsumed yet.
Author: Wenchen Fan <wenchen@databricks.com>
Closes #9346 from cloud-fan/cogroup and squashes the following commits:
9be67c8 [Wenchen Fan] SPARK-11393
Diffstat (limited to 'core')
0 files changed, 0 insertions, 0 deletions