diff options
| author | Holden Karau <holden@pigscanfly.ca> | 2015-09-18 13:47:14 -0700 |
|---|---|---|
| committer | Cheng Lian <lian@databricks.com> | 2015-09-18 13:47:14 -0700 |
| commit | 3a22b1004f527d54d399dd0225cd7f2f8ffad9c5 (patch) | |
| tree | 7240c9eb08b33e1a2e4a098a4de8e46af17ff2b3 /sql | |
| parent | c6f8135ee52202bd86adb090ab631e80330ea4df (diff) | |
| download | spark-3a22b1004f527d54d399dd0225cd7f2f8ffad9c5.tar.gz spark-3a22b1004f527d54d399dd0225cd7f2f8ffad9c5.tar.bz2 spark-3a22b1004f527d54d399dd0225cd7f2f8ffad9c5.zip | |
[SPARK-10449] [SQL] Don't merge decimal types with incompatable precision or scales
From JIRA: Schema merging should only handle struct fields. But currently we also reconcile decimal precision and scale information.
Author: Holden Karau <holden@pigscanfly.ca>
Closes #8634 from holdenk/SPARK-10449-dont-merge-different-precision.
Diffstat (limited to 'sql')
| -rw-r--r-- | sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala | 17 |
1 files changed, 13 insertions, 4 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala index b29cf22dcb..d6b436724b 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala @@ -373,10 +373,19 @@ object StructType extends AbstractDataType { StructType(newFields) case (DecimalType.Fixed(leftPrecision, leftScale), - DecimalType.Fixed(rightPrecision, rightScale)) => - DecimalType( - max(leftScale, rightScale) + max(leftPrecision - leftScale, rightPrecision - rightScale), - max(leftScale, rightScale)) + DecimalType.Fixed(rightPrecision, rightScale)) => + if ((leftPrecision == rightPrecision) && (leftScale == rightScale)) { + DecimalType(leftPrecision, leftScale) + } else if ((leftPrecision != rightPrecision) && (leftScale != rightScale)) { + throw new SparkException("Failed to merge Decimal Tpes with incompatible " + + s"precision $leftPrecision and $rightPrecision & scale $leftScale and $rightScale") + } else if (leftPrecision != rightPrecision) { + throw new SparkException("Failed to merge Decimal Tpes with incompatible " + + s"precision $leftPrecision and $rightPrecision") + } else { + throw new SparkException("Failed to merge Decimal Tpes with incompatible " + + s"scala $leftScale and $rightScale") + } case (leftUdt: UserDefinedType[_], rightUdt: UserDefinedType[_]) if leftUdt.userClass == rightUdt.userClass => leftUdt |