diff options
| author | Wenchen Fan <wenchen@databricks.com> | 2016-04-11 22:58:35 -0700 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2016-04-11 22:58:35 -0700 |
| commit | 52a801124f429ab133f9a3867c1da6ebd8fa7d4e (patch) | |
| tree | ec09e7b7bfa2c86207d65d0068251fa6e7852513 /streaming | |
| parent | 2d81ba542e12db65c2bd67357093244be9403102 (diff) | |
| download | spark-52a801124f429ab133f9a3867c1da6ebd8fa7d4e.tar.gz spark-52a801124f429ab133f9a3867c1da6ebd8fa7d4e.tar.bz2 spark-52a801124f429ab133f9a3867c1da6ebd8fa7d4e.zip | |
[SPARK-14554][SQL] disable whole stage codegen if there are too many input columns
## What changes were proposed in this pull request?
In https://github.com/apache/spark/pull/12047/files#diff-94a1f59bcc9b6758c4ca874652437634R529, we may split field expressions codes in `CreateExternalRow` to support wide table. However, the whole stage codegen framework doesn't support it, because the input for expressions is not always the input row, but can be `CodeGenContext.currentVars`, which doesn't work well with `CodeGenContext.splitExpressions`.
Actually we do have a check to guard against this cases, but it's incomplete, it only checks output fields.
This PR improves the whole stage codegen support check, to disable it if there are too many input fields, so that we can avoid splitting field expressions codes in `CreateExternalRow` for whole stage codegen.
TODO: Is it a better solution if we can make `CodeGenContext.currentVars` work well with `CodeGenContext.splitExpressions`?
## How was this patch tested?
new test in DatasetSuite.
Author: Wenchen Fan <wenchen@databricks.com>
Closes #12322 from cloud-fan/codegen.
Diffstat (limited to 'streaming')
0 files changed, 0 insertions, 0 deletions