diff options
| author | Jason White <jason.white@shopify.com> | 2016-10-03 14:12:03 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-10-03 14:12:03 -0700 |
| commit | 1f31bdaef670dd43999613deae3620f4ddcd1fbf (patch) | |
| tree | 0814b539f497beddf89f2421e90aa93d0bf83c6b /python/pyspark/java_gateway.py | |
| parent | 1dd68d3827133d203e85294405400b04904879e0 (diff) | |
| download | spark-1f31bdaef670dd43999613deae3620f4ddcd1fbf.tar.gz spark-1f31bdaef670dd43999613deae3620f4ddcd1fbf.tar.bz2 spark-1f31bdaef670dd43999613deae3620f4ddcd1fbf.zip | |
[SPARK-17679] [PYSPARK] remove unnecessary Py4J ListConverter patch
## What changes were proposed in this pull request?
This PR removes a patch on ListConverter from https://github.com/apache/spark/pull/5570, as it is no longer necessary. The underlying issue in Py4J https://github.com/bartdag/py4j/issues/160 was patched in https://github.com/bartdag/py4j/commit/224b94b6665e56a93a064073886e1d803a4969d2 and is present in 0.10.3, the version currently in use in Spark.
## How was this patch tested?
The original test added in https://github.com/apache/spark/pull/5570 remains.
Author: Jason White <jason.white@shopify.com>
Closes #15254 from JasonMWhite/remove_listconverter_patch.
Diffstat (limited to 'python/pyspark/java_gateway.py')
| -rw-r--r-- | python/pyspark/java_gateway.py | 9 |
1 files changed, 0 insertions, 9 deletions
diff --git a/python/pyspark/java_gateway.py b/python/pyspark/java_gateway.py index 527ca82d31..f76cadcf62 100644 --- a/python/pyspark/java_gateway.py +++ b/python/pyspark/java_gateway.py @@ -29,18 +29,9 @@ if sys.version >= '3': xrange = range from py4j.java_gateway import java_import, JavaGateway, GatewayClient -from py4j.java_collections import ListConverter - from pyspark.serializers import read_int -# patching ListConverter, or it will convert bytearray into Java ArrayList -def can_convert_list(self, obj): - return isinstance(obj, (list, tuple, xrange)) - -ListConverter.can_convert = can_convert_list - - def launch_gateway(): if "PYSPARK_GATEWAY_PORT" in os.environ: gateway_port = int(os.environ["PYSPARK_GATEWAY_PORT"]) |