diff options
author | Cheng Lian <lian.cs.zju@gmail.com> | 2014-05-24 20:42:01 -0700 |
---|---|---|
committer | Reynold Xin <rxin@apache.org> | 2014-05-24 20:42:01 -0700 |
commit | 5afe6af0b192ce7e908634992e8752537b1c4ed1 (patch) | |
tree | 74064440c9b3cdfe917fe5687d79bce8c2677cd7 /sql/hive | |
parent | 4e4831b8facc186cda6ef31040ccdeab48acbbb7 (diff) | |
download | spark-5afe6af0b192ce7e908634992e8752537b1c4ed1.tar.gz spark-5afe6af0b192ce7e908634992e8752537b1c4ed1.tar.bz2 spark-5afe6af0b192ce7e908634992e8752537b1c4ed1.zip |
[SPARK-1913][SQL] Bug fix: column pruning error in Parquet support
JIRA issue: [SPARK-1913](https://issues.apache.org/jira/browse/SPARK-1913)
When scanning Parquet tables, attributes referenced only in predicates that are pushed down are not passed to the `ParquetTableScan` operator and causes exception.
Author: Cheng Lian <lian.cs.zju@gmail.com>
Closes #863 from liancheng/spark-1913 and squashes the following commits:
f976b73 [Cheng Lian] Addessed the readability issue commented by @rxin
f5b257d [Cheng Lian] Added back comments deleted by mistake
ae60ab3 [Cheng Lian] [SPARK-1913] Attributes referenced only in predicates pushed down should remain in ParquetTableScan operator
Diffstat (limited to 'sql/hive')
-rw-r--r-- | sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala index b2157074a4..8b51957162 100644 --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala @@ -69,6 +69,7 @@ private[hive] trait HiveStrategies { pruneFilterProject( projectList, otherPredicates, + identity[Seq[Expression]], HiveTableScan(_, relation, pruningPredicates.reduceLeftOption(And))(hiveContext)) :: Nil case _ => Nil |