diff options
author | Davies Liu <davies.liu@gmail.com> | 2014-09-15 18:57:25 -0700 |
---|---|---|
committer | Josh Rosen <joshrosen@apache.org> | 2014-09-15 18:57:25 -0700 |
commit | da33acb8b681eca5e787d546fe922af76a151398 (patch) | |
tree | 1160991b887d207efc3cf3fcad786de8811570e2 /python/pyspark/context.py | |
parent | fdb302f49c021227026909bdcdade7496059013f (diff) | |
download | spark-da33acb8b681eca5e787d546fe922af76a151398.tar.gz spark-da33acb8b681eca5e787d546fe922af76a151398.tar.bz2 spark-da33acb8b681eca5e787d546fe922af76a151398.zip |
[SPARK-2951] [PySpark] support unpickle array.array for Python 2.6
Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite.
There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue.
I had send an PR to Pyrolite to fix it: https://github.com/irmen/Pyrolite/pull/11
Author: Davies Liu <davies.liu@gmail.com>
Closes #2365 from davies/pickle and squashes the following commits:
f44f771 [Davies Liu] enable tests about array
3908f5c [Davies Liu] Merge branch 'master' into pickle
c77c87b [Davies Liu] cleanup debugging code
60e4e2f [Davies Liu] support unpickle array.array for Python 2.6
Diffstat (limited to 'python/pyspark/context.py')
-rw-r--r-- | python/pyspark/context.py | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/python/pyspark/context.py b/python/pyspark/context.py index 3ab98e262d..ea28e8cd8c 100644 --- a/python/pyspark/context.py +++ b/python/pyspark/context.py @@ -214,6 +214,7 @@ class SparkContext(object): SparkContext._gateway = gateway or launch_gateway() SparkContext._jvm = SparkContext._gateway.jvm SparkContext._writeToFile = SparkContext._jvm.PythonRDD.writeToFile + SparkContext._jvm.SerDeUtil.initialize() if instance: if (SparkContext._active_spark_context and |