diff options
author | wm624@hotmail.com <wm624@hotmail.com> | 2016-09-02 01:47:17 -0700 |
---|---|---|
committer | Felix Cheung <felixcheung@apache.org> | 2016-09-02 01:47:17 -0700 |
commit | 0f30cdedbdb0d38e8c479efab6bb1c6c376206ff (patch) | |
tree | e5b05ff7584f77d551da0dfc2269a58d7fd1da43 /R/pkg/NAMESPACE | |
parent | 2ab8dbddaa31e4491b52eb0e495660ebbebfdb9e (diff) | |
download | spark-0f30cdedbdb0d38e8c479efab6bb1c6c376206ff.tar.gz spark-0f30cdedbdb0d38e8c479efab6bb1c6c376206ff.tar.bz2 spark-0f30cdedbdb0d38e8c479efab6bb1c6c376206ff.zip |
[SPARK-16883][SPARKR] SQL decimal type is not properly cast to number when collecting SparkDataFrame
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
registerTempTable(createDataFrame(iris), "iris")
str(collect(sql("select cast('1' as double) as x, cast('2' as decimal) as y from iris limit 5")))
'data.frame': 5 obs. of 2 variables:
$ x: num 1 1 1 1 1
$ y:List of 5
..$ : num 2
..$ : num 2
..$ : num 2
..$ : num 2
..$ : num 2
The problem is that spark returns `decimal(10, 0)` col type, instead of `decimal`. Thus, `decimal(10, 0)` is not handled correctly. It should be handled as "double".
As discussed in JIRA thread, we can have two potential fixes:
1). Scala side fix to add a new case when writing the object back; However, I can't use spark.sql.types._ in Spark core due to dependency issues. I don't find a way of doing type case match;
2). SparkR side fix: Add a helper function to check special type like `"decimal(10, 0)"` and replace it with `double`, which is PRIMITIVE type. This special helper is generic for adding new types handling in the future.
I open this PR to discuss pros and cons of both approaches. If we want to do Scala side fix, we need to find a way to match the case of DecimalType and StructType in Spark Core.
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
Manual test:
> str(collect(sql("select cast('1' as double) as x, cast('2' as decimal) as y from iris limit 5")))
'data.frame': 5 obs. of 2 variables:
$ x: num 1 1 1 1 1
$ y: num 2 2 2 2 2
R Unit tests
Author: wm624@hotmail.com <wm624@hotmail.com>
Closes #14613 from wangmiao1981/type.
Diffstat (limited to 'R/pkg/NAMESPACE')
0 files changed, 0 insertions, 0 deletions