diff options
| author | Reynold Xin <rxin@databricks.com> | 2015-07-27 22:51:15 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-07-27 22:51:15 -0700 |
| commit | 60f08c7c8775c0462b74bc65b41397be6eb24b6d (patch) | |
| tree | 7d693ca7b1fe05af39bc88942251fd24df1101cb /unsafe | |
| parent | 63a492b931765b1edd66624421d503f1927825ec (diff) | |
| download | spark-60f08c7c8775c0462b74bc65b41397be6eb24b6d.tar.gz spark-60f08c7c8775c0462b74bc65b41397be6eb24b6d.tar.bz2 spark-60f08c7c8775c0462b74bc65b41397be6eb24b6d.zip | |
[SPARK-9373][SQL] Support StructType in Tungsten projection
This pull request updates GenerateUnsafeProjection to support StructType. If an input struct type is backed already by an UnsafeRow, GenerateUnsafeProjection copies the bytes directly into its buffer space without any conversion. However, if the input is not an UnsafeRow, GenerateUnsafeProjection runs the code generated recursively to convert the input into an UnsafeRow and then copies it into the buffer space.
Also create a TungstenProject operator that projects data directly into UnsafeRow. Note that I'm not sure if this is the way we want to structure Unsafe+codegen operators, but we can defer that decision to follow-up pull requests.
Author: Reynold Xin <rxin@databricks.com>
Closes #7689 from rxin/tungsten-struct-type and squashes the following commits:
9162f42 [Reynold Xin] Support IntervalType in UnsafeRow's getter.
be9f377 [Reynold Xin] Fixed tests.
10c4b7c [Reynold Xin] Format generated code.
77e8d0e [Reynold Xin] Fixed NondeterministicSuite.
ac4951d [Reynold Xin] Yay.
ac203bf [Reynold Xin] More comments.
9f36216 [Reynold Xin] Updated comment.
6b781fe [Reynold Xin] Reset the change in DataFrameSuite.
525b95b [Reynold Xin] Merged with master, more documentation & test cases.
321859a [Reynold Xin] [SPARK-9373][SQL] Support StructType in Tungsten projection [WIP]
Diffstat (limited to 'unsafe')
0 files changed, 0 insertions, 0 deletions