aboutsummaryrefslogtreecommitdiff
path: root/sql/core/src/main
diff options
context:
space:
mode:
authorDB Tsai <dbt@netflix.com>2017-04-12 11:19:20 +0800
committerWenchen Fan <wenchen@databricks.com>2017-04-12 11:19:20 +0800
commit8ad63ee158815de5ffff7bf03cdf25aef312095f (patch)
tree6dffd720a03903f518d86b2e3c444966f9baad1b /sql/core/src/main
parentcde9e328484e4007aa6b505312d7cea5461a6eaf (diff)
downloadspark-8ad63ee158815de5ffff7bf03cdf25aef312095f.tar.gz
spark-8ad63ee158815de5ffff7bf03cdf25aef312095f.tar.bz2
spark-8ad63ee158815de5ffff7bf03cdf25aef312095f.zip
[SPARK-20291][SQL] NaNvl(FloatType, NullType) should not be cast to NaNvl(DoubleType, DoubleType)
## What changes were proposed in this pull request? `NaNvl(float value, null)` will be converted into `NaNvl(float value, Cast(null, DoubleType))` and finally `NaNvl(Cast(float value, DoubleType), Cast(null, DoubleType))`. This will cause mismatching in the output type when the input type is float. By adding extra rule in TypeCoercion can resolve this issue. ## How was this patch tested? unite tests. Please review http://spark.apache.org/contributing.html before opening a pull request. Author: DB Tsai <dbt@netflix.com> Closes #17606 from dbtsai/fixNaNvl.
Diffstat (limited to 'sql/core/src/main')
-rw-r--r--sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala3
1 files changed, 1 insertions, 2 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala
index 93d565d9fe..052d85ad33 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala
@@ -408,8 +408,7 @@ final class DataFrameNaFunctions private[sql](df: DataFrame) {
val quotedColName = "`" + col.name + "`"
val colValue = col.dataType match {
case DoubleType | FloatType =>
- // nanvl only supports these types
- nanvl(df.col(quotedColName), lit(null).cast(col.dataType))
+ nanvl(df.col(quotedColName), lit(null)) // nanvl only supports these types
case _ => df.col(quotedColName)
}
coalesce(colValue, lit(replacement).cast(col.dataType)).as(col.name)