diff options
| author | Dongjoon Hyun <dongjoon@apache.org> | 2017-03-15 15:01:16 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2017-03-15 15:01:16 -0700 |
| commit | 54a3697f1fb562ef9ed8fed9caffc62b84763049 (patch) | |
| tree | cb86b16a938da3f29d2028df7fed9c6488bfdcfd /sql/core | |
| parent | 97cc5e5a5555519d221d0ca78645dde9bb8ea40b (diff) | |
| download | spark-54a3697f1fb562ef9ed8fed9caffc62b84763049.tar.gz spark-54a3697f1fb562ef9ed8fed9caffc62b84763049.tar.bz2 spark-54a3697f1fb562ef9ed8fed9caffc62b84763049.zip | |
[MINOR][CORE] Fix a info message of `prunePartitions`
## What changes were proposed in this pull request?
`PrunedInMemoryFileIndex.prunePartitions` shows `pruned NaN% partitions` for the following case.
```scala
scala> Seq.empty[(String, String)].toDF("a", "p").write.partitionBy("p").saveAsTable("t1")
scala> sc.setLogLevel("INFO")
scala> spark.table("t1").filter($"p" === "1").select($"a").show
...
17/03/13 00:33:04 INFO PrunedInMemoryFileIndex: Selected 0 partitions out of 0, pruned NaN% partitions.
```
After this PR, the message looks like this.
```scala
17/03/15 10:39:48 INFO PrunedInMemoryFileIndex: Selected 0 partitions out of 0, pruned 0 partitions.
```
## How was this patch tested?
Pass the Jenkins with the existing tests.
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #17273 from dongjoon-hyun/SPARK-EMPTY-PARTITION.
Diffstat (limited to 'sql/core')
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala index a5fa8b3f93..db8bbc52aa 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala @@ -186,7 +186,8 @@ abstract class PartitioningAwareFileIndex( val total = partitions.length val selectedSize = selected.length val percentPruned = (1 - selectedSize.toDouble / total.toDouble) * 100 - s"Selected $selectedSize partitions out of $total, pruned $percentPruned% partitions." + s"Selected $selectedSize partitions out of $total, " + + s"pruned ${if (total == 0) "0" else s"$percentPruned%"} partitions." } selected |