diff options
author | Wenchen Fan <wenchen@databricks.com> | 2016-05-11 11:16:05 -0700 |
---|---|---|
committer | Davies Liu <davies.liu@gmail.com> | 2016-05-11 11:16:05 -0700 |
commit | d8935db5ecb7c959585411da9bf1e9a9c4d5cb37 (patch) | |
tree | b57911a28a577f06a02f385ac9c7ec6d45ec2417 /docs/_plugins/include_example.rb | |
parent | e1576478bde2b9146bdbd4f2bae20a4011b20229 (diff) | |
download | spark-d8935db5ecb7c959585411da9bf1e9a9c4d5cb37.tar.gz spark-d8935db5ecb7c959585411da9bf1e9a9c4d5cb37.tar.bz2 spark-d8935db5ecb7c959585411da9bf1e9a9c4d5cb37.zip |
[SPARK-15241] [SPARK-15242] [SQL] fix 2 decimal-related issues in RowEncoder
## What changes were proposed in this pull request?
SPARK-15241: We now support java decimal and catalyst decimal in external row, it makes sense to also support scala decimal.
SPARK-15242: This is a long-standing bug, and is exposed after https://github.com/apache/spark/pull/12364, which eliminate the `If` expression if the field is not nullable:
```
val fieldValue = serializerFor(
GetExternalRowField(inputObject, i, externalDataTypeForInput(f.dataType)),
f.dataType)
if (f.nullable) {
If(
Invoke(inputObject, "isNullAt", BooleanType, Literal(i) :: Nil),
Literal.create(null, f.dataType),
fieldValue)
} else {
fieldValue
}
```
Previously, we always use `DecimalType.SYSTEM_DEFAULT` as the output type of converted decimal field, which is wrong as it doesn't match the real decimal type. However, it works well because we always put converted field into `If` expression to do the null check, and `If` use its `trueValue`'s data type as its output type.
Now if we have a not nullable decimal field, then the converted field's output type will be `DecimalType.SYSTEM_DEFAULT`, and we will write wrong data into unsafe row.
The fix is simple, just use the given decimal type as the output type of converted decimal field.
These 2 issues was found at https://github.com/apache/spark/pull/13008
## How was this patch tested?
new tests in RowEncoderSuite
Author: Wenchen Fan <wenchen@databricks.com>
Closes #13019 from cloud-fan/encoder-decimal.
Diffstat (limited to 'docs/_plugins/include_example.rb')
0 files changed, 0 insertions, 0 deletions