diff options
| author | Davies Liu <davies@databricks.com> | 2016-06-17 00:34:33 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-06-17 00:34:33 -0700 |
| commit | ef43b4ed87894982678fcc6f2c61cf1487ee9e14 (patch) | |
| tree | 7aa82f972352953746fe09fcf38defa3cee5de9b /sql | |
| parent | e5d703bca85c65ce329b1e202283cfa35d109146 (diff) | |
| download | spark-ef43b4ed87894982678fcc6f2c61cf1487ee9e14.tar.gz spark-ef43b4ed87894982678fcc6f2c61cf1487ee9e14.tar.bz2 spark-ef43b4ed87894982678fcc6f2c61cf1487ee9e14.zip | |
[SPARK-15811][SQL] fix the Python UDF in Scala 2.10
## What changes were proposed in this pull request?
Iterator can't be serialized in Scala 2.10, we should force it into a array to make sure that .
## How was this patch tested?
Build with Scala 2.10 and ran all the Python unit tests manually (will be covered by a jenkins build).
Author: Davies Liu <davies@databricks.com>
Closes #13717 from davies/fix_udf_210.
Diffstat (limited to 'sql')
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala index 668470ee6a..87583c8234 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala @@ -132,7 +132,7 @@ private[spark] object ExtractPythonUDFs extends Rule[SparkPlan] { val validUdfs = udfs.filter { case udf => // Check to make sure that the UDF can be evaluated with only the input of this child. udf.references.subsetOf(child.outputSet) - } + }.toArray // Turn it into an array since iterators cannot be serialized in Scala 2.10 if (validUdfs.nonEmpty) { val resultAttrs = udfs.zipWithIndex.map { case (u, i) => AttributeReference(s"pythonUDF$i", u.dataType)() |