diff options
| author | Kazuaki Ishizaki <ishizaki@jp.ibm.com> | 2016-12-02 12:30:13 +0800 |
|---|---|---|
| committer | Wenchen Fan <wenchen@databricks.com> | 2016-12-02 12:30:13 +0800 |
| commit | 38b9e69623c14a675b14639e8291f5d29d2a0bc3 (patch) | |
| tree | 7dabeeb22f97923554f9fa155c6e7e22733ad060 /conf | |
| parent | 70c5549ee9588228d18a7b405c977cf591e2efd4 (diff) | |
| download | spark-38b9e69623c14a675b14639e8291f5d29d2a0bc3.tar.gz spark-38b9e69623c14a675b14639e8291f5d29d2a0bc3.tar.bz2 spark-38b9e69623c14a675b14639e8291f5d29d2a0bc3.zip | |
[SPARK-18284][SQL] Make ExpressionEncoder.serializer.nullable precise
## What changes were proposed in this pull request?
This PR makes `ExpressionEncoder.serializer.nullable` for flat encoder for a primitive type `false`. Since it is `true` for now, it is too conservative.
While `ExpressionEncoder.schema` has correct information (e.g. `<IntegerType, false>`), `serializer.head.nullable` of `ExpressionEncoder`, which got from `encoderFor[T]`, is always false. It is too conservative.
This is accomplished by checking whether a type is one of primitive types. If it is `true`, `nullable` should be `false`.
## How was this patch tested?
Added new tests for encoder and dataframe
Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Closes #15780 from kiszk/SPARK-18284.
Diffstat (limited to 'conf')
0 files changed, 0 insertions, 0 deletions