diff options
| author | Davies Liu <davies@databricks.com> | 2016-04-04 10:56:26 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-04-04 10:56:26 -0700 |
| commit | 5743c6476dbef50852b7f9873112a2d299966ebd (patch) | |
| tree | 63c8a3682266a0eb2fbcf83ab298d2082bf4bdca /sql/hive/src/main | |
| parent | 855ed44ed31210d2001d7ce67c8fa99f8416edd3 (diff) | |
| download | spark-5743c6476dbef50852b7f9873112a2d299966ebd.tar.gz spark-5743c6476dbef50852b7f9873112a2d299966ebd.tar.bz2 spark-5743c6476dbef50852b7f9873112a2d299966ebd.zip | |
[SPARK-12981] [SQL] extract Pyhton UDF in physical plan
## What changes were proposed in this pull request?
Currently we extract Python UDFs into a special logical plan EvaluatePython in analyzer, But EvaluatePython is not part of catalyst, many rules have no knowledge of it , which will break many things (for example, filter push down or column pruning).
We should treat Python UDFs as normal expressions, until we want to evaluate in physical plan, we could extract them in end of optimizer, or physical plan.
This PR extract Python UDFs in physical plan.
Closes #10935
## How was this patch tested?
Added regression tests.
Author: Davies Liu <davies@databricks.com>
Closes #12127 from davies/py_udf.
Diffstat (limited to 'sql/hive/src/main')
| -rw-r--r-- | sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala | 1 |
1 files changed, 0 insertions, 1 deletions
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala index ff40c366c8..829afa8432 100644 --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala @@ -60,7 +60,6 @@ private[hive] class HiveSessionState(ctx: HiveContext) extends SessionState(ctx) catalog.OrcConversions :: catalog.CreateTables :: catalog.PreInsertionCasts :: - python.ExtractPythonUDFs :: PreInsertCastAndRename :: DataSourceAnalysis :: (if (conf.runSQLOnFile) new ResolveDataSource(ctx) :: Nil else Nil) |