aboutsummaryrefslogtreecommitdiff
path: root/R/pkg/NAMESPACE
diff options
context:
space:
mode:
authorwm624@hotmail.com <wm624@hotmail.com>2016-09-02 01:47:17 -0700
committerFelix Cheung <felixcheung@apache.org>2016-09-02 01:47:17 -0700
commit0f30cdedbdb0d38e8c479efab6bb1c6c376206ff (patch)
treee5b05ff7584f77d551da0dfc2269a58d7fd1da43 /R/pkg/NAMESPACE
parent2ab8dbddaa31e4491b52eb0e495660ebbebfdb9e (diff)
downloadspark-0f30cdedbdb0d38e8c479efab6bb1c6c376206ff.tar.gz
spark-0f30cdedbdb0d38e8c479efab6bb1c6c376206ff.tar.bz2
spark-0f30cdedbdb0d38e8c479efab6bb1c6c376206ff.zip
[SPARK-16883][SPARKR] SQL decimal type is not properly cast to number when collecting SparkDataFrame
## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) registerTempTable(createDataFrame(iris), "iris") str(collect(sql("select cast('1' as double) as x, cast('2' as decimal) as y from iris limit 5"))) 'data.frame': 5 obs. of 2 variables: $ x: num 1 1 1 1 1 $ y:List of 5 ..$ : num 2 ..$ : num 2 ..$ : num 2 ..$ : num 2 ..$ : num 2 The problem is that spark returns `decimal(10, 0)` col type, instead of `decimal`. Thus, `decimal(10, 0)` is not handled correctly. It should be handled as "double". As discussed in JIRA thread, we can have two potential fixes: 1). Scala side fix to add a new case when writing the object back; However, I can't use spark.sql.types._ in Spark core due to dependency issues. I don't find a way of doing type case match; 2). SparkR side fix: Add a helper function to check special type like `"decimal(10, 0)"` and replace it with `double`, which is PRIMITIVE type. This special helper is generic for adding new types handling in the future. I open this PR to discuss pros and cons of both approaches. If we want to do Scala side fix, we need to find a way to match the case of DecimalType and StructType in Spark Core. ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) Manual test: > str(collect(sql("select cast('1' as double) as x, cast('2' as decimal) as y from iris limit 5"))) 'data.frame': 5 obs. of 2 variables: $ x: num 1 1 1 1 1 $ y: num 2 2 2 2 2 R Unit tests Author: wm624@hotmail.com <wm624@hotmail.com> Closes #14613 from wangmiao1981/type.
Diffstat (limited to 'R/pkg/NAMESPACE')
0 files changed, 0 insertions, 0 deletions