diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2015-07-30 17:38:48 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-07-30 17:38:48 -0700 |
| commit | 3c66ff727d4b47220e1ff363cea215189ed64f36 (patch) | |
| tree | 51e2f06cb4916d1104a8c846d6ee14889931ed39 /streaming/src | |
| parent | 65fa4181c35135080870c1e4c1f904ada3a8cf59 (diff) | |
| download | spark-3c66ff727d4b47220e1ff363cea215189ed64f36.tar.gz spark-3c66ff727d4b47220e1ff363cea215189ed64f36.tar.bz2 spark-3c66ff727d4b47220e1ff363cea215189ed64f36.zip | |
[SPARK-9489] Remove unnecessary compatibility and requirements checks from Exchange
While reviewing yhuai's patch for SPARK-2205 (#7773), I noticed that Exchange's `compatible` check may be incorrectly returning `false` in many cases. As far as I know, this is not actually a problem because the `compatible`, `meetsRequirements`, and `needsAnySort` checks are serving only as short-circuit performance optimizations that are not necessary for correctness.
In order to reduce code complexity, I think that we should remove these checks and unconditionally rewrite the operator's children. This should be safe because we rewrite the tree in a single bottom-up pass.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #7807 from JoshRosen/SPARK-9489 and squashes the following commits:
9d76ce9 [Josh Rosen] [SPARK-9489] Remove compatibleWith, meetsRequirements, and needsAnySort checks from Exchange
Diffstat (limited to 'streaming/src')
0 files changed, 0 insertions, 0 deletions