diff options
| author | Huaxin Gao <huaxing@us.ibm.com> | 2016-12-10 22:41:40 +0800 |
|---|---|---|
| committer | Wenchen Fan <wenchen@databricks.com> | 2016-12-10 22:41:40 +0800 |
| commit | c5172568b59b4cf1d3dc7ed8c17a9bea2ea2ab79 (patch) | |
| tree | be3605d08b48437dc41b3524ccd9bc351d895c2f /sql/catalyst/src | |
| parent | 63c9159870ee274c68e24360594ca01d476b9ace (diff) | |
| download | spark-c5172568b59b4cf1d3dc7ed8c17a9bea2ea2ab79.tar.gz spark-c5172568b59b4cf1d3dc7ed8c17a9bea2ea2ab79.tar.bz2 spark-c5172568b59b4cf1d3dc7ed8c17a9bea2ea2ab79.zip | |
[SPARK-17460][SQL] Make sure sizeInBytes in Statistics will not overflow
## What changes were proposed in this pull request?
1. In SparkStrategies.canBroadcast, I will add the check plan.statistics.sizeInBytes >= 0
2. In LocalRelations.statistics, when calculate the statistics, I will change the size to BigInt so it won't overflow.
## How was this patch tested?
I will add a test case to make sure the statistics.sizeInBytes won't overflow.
Author: Huaxin Gao <huaxing@us.ibm.com>
Closes #16175 from huaxingao/spark-17460.
Diffstat (limited to 'sql/catalyst/src')
| -rw-r--r-- | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala index 890865d177..91633f5124 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala @@ -75,7 +75,8 @@ case class LocalRelation(output: Seq[Attribute], data: Seq[InternalRow] = Nil) } override lazy val statistics = - Statistics(sizeInBytes = output.map(_.dataType.defaultSize).sum * data.length) + Statistics(sizeInBytes = + (output.map(n => BigInt(n.dataType.defaultSize))).sum * data.length) def toSQL(inlineTableName: String): String = { require(data.nonEmpty) |