diff options
| author | Davies Liu <davies@databricks.com> | 2015-08-10 13:52:18 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-08-10 13:52:18 -0700 |
| commit | fe2fb7fb7189d183a4273ad27514af4b6b461f26 (patch) | |
| tree | c9983b34e92572f33bca638e5fe2dfc5af3e6c86 /launcher | |
| parent | 40ed2af587cedadc6e5249031857a922b3b234ca (diff) | |
| download | spark-fe2fb7fb7189d183a4273ad27514af4b6b461f26.tar.gz spark-fe2fb7fb7189d183a4273ad27514af4b6b461f26.tar.bz2 spark-fe2fb7fb7189d183a4273ad27514af4b6b461f26.zip | |
[SPARK-9620] [SQL] generated UnsafeProjection should support many columns or large exressions
Currently, generated UnsafeProjection can reach 64k byte code limit of Java. This patch will split the generated expressions into multiple functions, to avoid the limitation.
After this patch, we can work well with table that have up to 64k columns (hit max number of constants limit in Java), it should be enough in practice.
cc rxin
Author: Davies Liu <davies@databricks.com>
Closes #8044 from davies/wider_table and squashes the following commits:
9192e6c [Davies Liu] fix generated safe projection
d1ef81a [Davies Liu] fix failed tests
737b3d3 [Davies Liu] Merge branch 'master' of github.com:apache/spark into wider_table
ffcd132 [Davies Liu] address comments
1b95be4 [Davies Liu] put the generated class into sql package
77ed72d [Davies Liu] address comments
4518e17 [Davies Liu] Merge branch 'master' of github.com:apache/spark into wider_table
75ccd01 [Davies Liu] Merge branch 'master' of github.com:apache/spark into wider_table
495e932 [Davies Liu] support wider table with more than 1k columns for generated projections
Diffstat (limited to 'launcher')
0 files changed, 0 insertions, 0 deletions