diff options
author | Liang-Chi Hsieh <simonh@tw.ibm.com> | 2016-06-10 18:23:59 -0700 |
---|---|---|
committer | Cheng Lian <lian@databricks.com> | 2016-06-10 18:23:59 -0700 |
commit | bba5d7999f7b3ae9d816ea552ba9378fea1615a6 (patch) | |
tree | bf303da6296982bd924fb057a4c0a2306d6b68fa /sql/catalyst | |
parent | 54f758b5fc60ecb0da6b191939a72ef5829be38c (diff) | |
download | spark-bba5d7999f7b3ae9d816ea552ba9378fea1615a6.tar.gz spark-bba5d7999f7b3ae9d816ea552ba9378fea1615a6.tar.bz2 spark-bba5d7999f7b3ae9d816ea552ba9378fea1615a6.zip |
[SPARK-15639][SQL] Try to push down filter at RowGroups level for parquet reader
## What changes were proposed in this pull request?
The base class `SpecificParquetRecordReaderBase` used for vectorized parquet reader will try to get pushed-down filters from the given configuration. This pushed-down filters are used for RowGroups-level filtering. However, we don't set up the filters to push down into the configuration. In other words, the filters are not actually pushed down to do RowGroups-level filtering. This patch is to fix this and tries to set up the filters for pushing down to configuration for the reader.
## How was this patch tested?
Existing tests should be passed.
Author: Liang-Chi Hsieh <simonh@tw.ibm.com>
Closes #13371 from viirya/vectorized-reader-push-down-filter.
Diffstat (limited to 'sql/catalyst')
-rw-r--r-- | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala index 306a99d5a3..c06a1ea356 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala @@ -292,6 +292,14 @@ case class AttributeReference( } } + def withMetadata(newMetadata: Metadata): AttributeReference = { + if (metadata == newMetadata) { + this + } else { + AttributeReference(name, dataType, nullable, newMetadata)(exprId, qualifier, isGenerated) + } + } + override protected final def otherCopyArgs: Seq[AnyRef] = { exprId :: qualifier :: isGenerated :: Nil } |