diff options
| author | Liang-Chi Hsieh <viirya@appier.com> | 2015-09-17 09:21:21 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2015-09-17 09:21:21 -0700 |
| commit | 2a508df20d03b3d4a3c05b65fb02d849bc080ef9 (patch) | |
| tree | b3d6e2fcd02aad7e10584c1ac0edd42a005d8ebc /sql | |
| parent | e51345e1e04e439827a07c95887d14ba38333057 (diff) | |
| download | spark-2a508df20d03b3d4a3c05b65fb02d849bc080ef9.tar.gz spark-2a508df20d03b3d4a3c05b65fb02d849bc080ef9.tar.bz2 spark-2a508df20d03b3d4a3c05b65fb02d849bc080ef9.zip | |
[SPARK-10459] [SQL] Do not need to have ConvertToSafe for PythonUDF
JIRA: https://issues.apache.org/jira/browse/SPARK-10459
As mentioned in the JIRA, `PythonUDF` actually could process `UnsafeRow`.
Specially, the rows in `childResults` in `BatchPythonEvaluation` will be projected to a `MutableRow`. So I think we can enable `canProcessUnsafeRows` for `BatchPythonEvaluation` and get rid of redundant `ConvertToSafe`.
Author: Liang-Chi Hsieh <viirya@appier.com>
Closes #8616 from viirya/pyudf-unsafe.
Diffstat (limited to 'sql')
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala index 5a58d846ad..d0411da6fd 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala @@ -337,6 +337,10 @@ case class BatchPythonEvaluation(udf: PythonUDF, output: Seq[Attribute], child: def children: Seq[SparkPlan] = child :: Nil + override def outputsUnsafeRows: Boolean = false + override def canProcessUnsafeRows: Boolean = true + override def canProcessSafeRows: Boolean = true + protected override def doExecute(): RDD[InternalRow] = { val childResults = child.execute().map(_.copy()) |