aboutsummaryrefslogtreecommitdiff
path: root/sql/catalyst
diff options
context:
space:
mode:
authorLiang-Chi Hsieh <simonh@tw.ibm.com>2016-06-10 18:23:59 -0700
committerCheng Lian <lian@databricks.com>2016-06-10 18:23:59 -0700
commitbba5d7999f7b3ae9d816ea552ba9378fea1615a6 (patch)
treebf303da6296982bd924fb057a4c0a2306d6b68fa /sql/catalyst
parent54f758b5fc60ecb0da6b191939a72ef5829be38c (diff)
downloadspark-bba5d7999f7b3ae9d816ea552ba9378fea1615a6.tar.gz
spark-bba5d7999f7b3ae9d816ea552ba9378fea1615a6.tar.bz2
spark-bba5d7999f7b3ae9d816ea552ba9378fea1615a6.zip
[SPARK-15639][SQL] Try to push down filter at RowGroups level for parquet reader
## What changes were proposed in this pull request? The base class `SpecificParquetRecordReaderBase` used for vectorized parquet reader will try to get pushed-down filters from the given configuration. This pushed-down filters are used for RowGroups-level filtering. However, we don't set up the filters to push down into the configuration. In other words, the filters are not actually pushed down to do RowGroups-level filtering. This patch is to fix this and tries to set up the filters for pushing down to configuration for the reader. ## How was this patch tested? Existing tests should be passed. Author: Liang-Chi Hsieh <simonh@tw.ibm.com> Closes #13371 from viirya/vectorized-reader-push-down-filter.
Diffstat (limited to 'sql/catalyst')
-rw-r--r--sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala8
1 files changed, 8 insertions, 0 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
index 306a99d5a3..c06a1ea356 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
@@ -292,6 +292,14 @@ case class AttributeReference(
}
}
+ def withMetadata(newMetadata: Metadata): AttributeReference = {
+ if (metadata == newMetadata) {
+ this
+ } else {
+ AttributeReference(name, dataType, nullable, newMetadata)(exprId, qualifier, isGenerated)
+ }
+ }
+
override protected final def otherCopyArgs: Seq[AnyRef] = {
exprId :: qualifier :: isGenerated :: Nil
}