diff options
| author | Josh Rosen <joshrosen@apache.org> | 2014-08-18 20:42:19 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@apache.org> | 2014-08-18 20:42:32 -0700 |
| commit | 3a03259a0421b08269a3b23cab2bdb4f9108f5c5 (patch) | |
| tree | bb214dd19b374366de7a6447faa429a38e75b4e5 /python/pyspark/worker.py | |
| parent | 7d069bf0c57b75b53b449fcc51cf7fd616f8686d (diff) | |
| download | spark-3a03259a0421b08269a3b23cab2bdb4f9108f5c5.tar.gz spark-3a03259a0421b08269a3b23cab2bdb4f9108f5c5.tar.bz2 spark-3a03259a0421b08269a3b23cab2bdb4f9108f5c5.zip | |
[SPARK-3114] [PySpark] Fix Python UDFs in Spark SQL.
This fixes SPARK-3114, an issue where we inadvertently broke Python UDFs in Spark SQL.
This PR modifiers the test runner script to always run the PySpark SQL tests, irrespective of whether SparkSQL itself has been modified. It also includes Davies' fix for the bug.
Closes #2026.
Author: Josh Rosen <joshrosen@apache.org>
Author: Davies Liu <davies.liu@gmail.com>
Closes #2027 from JoshRosen/pyspark-sql-fix and squashes the following commits:
9af2708 [Davies Liu] bugfix: disable compression of command
0d8d3a4 [Josh Rosen] Always run Python Spark SQL tests.
(cherry picked from commit 1f1819b20f887b487557c31e54b8bcd95b582dc6)
Signed-off-by: Josh Rosen <joshrosen@apache.org>
Diffstat (limited to 'python/pyspark/worker.py')
| -rw-r--r-- | python/pyspark/worker.py | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py index 77a9c4a0e0..6805063e06 100644 --- a/python/pyspark/worker.py +++ b/python/pyspark/worker.py @@ -72,7 +72,7 @@ def main(infile, outfile): value = ser._read_with_length(infile) _broadcastRegistry[bid] = Broadcast(bid, value) - command = ser._read_with_length(infile) + command = pickleSer._read_with_length(infile) (func, deserializer, serializer) = command init_time = time.time() iterator = deserializer.load_stream(infile) |