diff options
| author | Wenchen Fan <wenchen@databricks.com> | 2016-12-05 11:37:13 -0800 |
|---|---|---|
| committer | Herman van Hovell <hvanhovell@databricks.com> | 2016-12-05 11:37:13 -0800 |
| commit | 01a7d33d0851d82fd1bb477a58d9925fe8d727d8 (patch) | |
| tree | c8145bdf2efb7e300400be96963391655ac1c662 /docs/running-on-yarn.md | |
| parent | 246012859f0ed5248809a2e00e8355fbdaa8beb5 (diff) | |
| download | spark-01a7d33d0851d82fd1bb477a58d9925fe8d727d8.tar.gz spark-01a7d33d0851d82fd1bb477a58d9925fe8d727d8.tar.bz2 spark-01a7d33d0851d82fd1bb477a58d9925fe8d727d8.zip | |
[SPARK-18711][SQL] should disable subexpression elimination for LambdaVariable
## What changes were proposed in this pull request?
This is kind of a long-standing bug, it's hidden until https://github.com/apache/spark/pull/15780 , which may add `AssertNotNull` on top of `LambdaVariable` and thus enables subexpression elimination.
However, subexpression elimination will evaluate the common expressions at the beginning, which is invalid for `LambdaVariable`. `LambdaVariable` usually represents loop variable, which can't be evaluated ahead of the loop.
This PR skips expressions containing `LambdaVariable` when doing subexpression elimination.
## How was this patch tested?
updated test in `DatasetAggregatorSuite`
Author: Wenchen Fan <wenchen@databricks.com>
Closes #16143 from cloud-fan/aggregator.
Diffstat (limited to 'docs/running-on-yarn.md')
0 files changed, 0 insertions, 0 deletions