aboutsummaryrefslogtreecommitdiff
path: root/sql
diff options
context:
space:
mode:
authorLiang-Chi Hsieh <viirya@gmail.com>2015-06-30 08:17:24 -0700
committerDavies Liu <davies@databricks.com>2015-06-30 08:17:24 -0700
commita48e61915354d33fb98944a8eb5a5d48dd102041 (patch)
tree0d588a32cb5f84fec90dbb14b7fc554b211c6655 /sql
parent865a834e51ac3074811a11fd99a36d942f7f7de8 (diff)
downloadspark-a48e61915354d33fb98944a8eb5a5d48dd102041.tar.gz
spark-a48e61915354d33fb98944a8eb5a5d48dd102041.tar.bz2
spark-a48e61915354d33fb98944a8eb5a5d48dd102041.zip
[SPARK-8680] [SQL] Slightly improve PropagateTypes
JIRA: https://issues.apache.org/jira/browse/SPARK-8680 This PR slightly improve `PropagateTypes` in `HiveTypeCoercion`. It moves `q.inputSet` outside `q transformExpressions` instead calling `inputSet` multiple times. It also builds a map of attributes for looking attribute easily. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #7087 from viirya/improve_propagatetypes and squashes the following commits: 5c314c1 [Liang-Chi Hsieh] For comments. 913f6ad [Liang-Chi Hsieh] Slightly improve PropagateTypes.
Diffstat (limited to 'sql')
-rw-r--r--sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala30
1 files changed, 16 insertions, 14 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
index c3d68197d6..e525ad623f 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
@@ -131,20 +131,22 @@ trait HiveTypeCoercion {
// Don't propagate types from unresolved children.
case q: LogicalPlan if !q.childrenResolved => q
- case q: LogicalPlan => q transformExpressions {
- case a: AttributeReference =>
- q.inputSet.find(_.exprId == a.exprId) match {
- // This can happen when a Attribute reference is born in a non-leaf node, for example
- // due to a call to an external script like in the Transform operator.
- // TODO: Perhaps those should actually be aliases?
- case None => a
- // Leave the same if the dataTypes match.
- case Some(newType) if a.dataType == newType.dataType => a
- case Some(newType) =>
- logDebug(s"Promoting $a to $newType in ${q.simpleString}}")
- newType
- }
- }
+ case q: LogicalPlan =>
+ val inputMap = q.inputSet.toSeq.map(a => (a.exprId, a)).toMap
+ q transformExpressions {
+ case a: AttributeReference =>
+ inputMap.get(a.exprId) match {
+ // This can happen when a Attribute reference is born in a non-leaf node, for example
+ // due to a call to an external script like in the Transform operator.
+ // TODO: Perhaps those should actually be aliases?
+ case None => a
+ // Leave the same if the dataTypes match.
+ case Some(newType) if a.dataType == newType.dataType => a
+ case Some(newType) =>
+ logDebug(s"Promoting $a to $newType in ${q.simpleString}}")
+ newType
+ }
+ }
}
}