diff options
| author | Davies Liu <davies@databricks.com> | 2015-07-29 22:30:49 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2015-07-29 22:30:49 -0700 |
| commit | e044705b4402f86d0557ecd146f3565388c7eeb4 (patch) | |
| tree | f8db4937fe17a3c9fdb651f605df057aecf9d597 /python/pyspark/shuffle.py | |
| parent | f5dd11339fc9a6d11350f63beeca7c14aec169b1 (diff) | |
| download | spark-e044705b4402f86d0557ecd146f3565388c7eeb4.tar.gz spark-e044705b4402f86d0557ecd146f3565388c7eeb4.tar.bz2 spark-e044705b4402f86d0557ecd146f3565388c7eeb4.zip | |
[SPARK-9116] [SQL] [PYSPARK] support Python only UDT in __main__
Also we could create a Python UDT without having a Scala one, it's important for Python users.
cc mengxr JoshRosen
Author: Davies Liu <davies@databricks.com>
Closes #7453 from davies/class_in_main and squashes the following commits:
4dfd5e1 [Davies Liu] add tests for Python and Scala UDT
793d9b2 [Davies Liu] Merge branch 'master' of github.com:apache/spark into class_in_main
dc65f19 [Davies Liu] address comment
a9a3c40 [Davies Liu] Merge branch 'master' of github.com:apache/spark into class_in_main
a86e1fc [Davies Liu] fix serialization
ad528ba [Davies Liu] Merge branch 'master' of github.com:apache/spark into class_in_main
63f52ef [Davies Liu] fix pylint check
655b8a9 [Davies Liu] Merge branch 'master' of github.com:apache/spark into class_in_main
316a394 [Davies Liu] support Python UDT with UTF
0bcb3ef [Davies Liu] fix bug in mllib
de986d6 [Davies Liu] fix test
83d65ac [Davies Liu] fix bug in StructType
55bb86e [Davies Liu] support Python UDT in __main__ (without Scala one)
Diffstat (limited to 'python/pyspark/shuffle.py')
| -rw-r--r-- | python/pyspark/shuffle.py | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/python/pyspark/shuffle.py b/python/pyspark/shuffle.py index 8fb71bac64..b8118bdb7c 100644 --- a/python/pyspark/shuffle.py +++ b/python/pyspark/shuffle.py @@ -606,7 +606,7 @@ class ExternalList(object): if not os.path.exists(d): os.makedirs(d) p = os.path.join(d, str(id(self))) - self._file = open(p, "wb+", 65536) + self._file = open(p, "w+b", 65536) self._ser = BatchedSerializer(CompressedSerializer(PickleSerializer()), 1024) os.unlink(p) |