diff options
| author | Liang-Chi Hsieh <viirya@gmail.com> | 2015-06-07 15:33:48 +0800 |
|---|---|---|
| committer | Cheng Lian <lian@databricks.com> | 2015-06-07 15:35:43 +0800 |
| commit | b4d54417e557eecafaa209f4842656c311797ab4 (patch) | |
| tree | 28c7c6678f6b2c6bae142843c6afac81b7313886 /sql | |
| parent | 9d1f4d66bd8bb899ae7beaac0a3e0be3d9fb9886 (diff) | |
| download | spark-b4d54417e557eecafaa209f4842656c311797ab4.tar.gz spark-b4d54417e557eecafaa209f4842656c311797ab4.tar.bz2 spark-b4d54417e557eecafaa209f4842656c311797ab4.zip | |
[SPARK-8141] [SQL] Precompute datatypes for partition columns and reuse it
JIRA: https://issues.apache.org/jira/browse/SPARK-8141
Author: Liang-Chi Hsieh <viirya@gmail.com>
Closes #6687 from viirya/reuse_partition_column_types and squashes the following commits:
dab0688 [Liang-Chi Hsieh] Reuse partitionColumnTypes.
(cherry picked from commit 26d07f1ece4174788b0bcdc338a14d0bbc0e3602)
Signed-off-by: Cheng Lian <lian@databricks.com>
Diffstat (limited to 'sql')
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala b/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala index f5bd2d2941..25887ba9a1 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala @@ -435,8 +435,9 @@ abstract class HadoopFsRelation private[sql](maybePartitionSpec: Option[Partitio // partition values. userDefinedPartitionColumns.map { partitionSchema => val spec = discoverPartitions() + val partitionColumnTypes = spec.partitionColumns.map(_.dataType) val castedPartitions = spec.partitions.map { case p @ Partition(values, path) => - val literals = values.toSeq.zip(spec.partitionColumns.map(_.dataType)).map { + val literals = values.toSeq.zip(partitionColumnTypes).map { case (value, dataType) => Literal.create(value, dataType) } val castedValues = partitionSchema.zip(literals).map { case (field, literal) => |