diff options
| author | Bryan Cutler <cutlerb@gmail.com> | 2016-10-11 08:29:52 +0200 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-10-11 08:29:52 +0200 |
| commit | 658c7147f5bf637f36e8c66b9207d94b1e7c74c5 (patch) | |
| tree | 82507f1f189d75e3c816967bdaca513c5aa4518f /python | |
| parent | 19401a203b441e3355f0d3fc3fd062b6d5bdee1f (diff) | |
| download | spark-658c7147f5bf637f36e8c66b9207d94b1e7c74c5.tar.gz spark-658c7147f5bf637f36e8c66b9207d94b1e7c74c5.tar.bz2 spark-658c7147f5bf637f36e8c66b9207d94b1e7c74c5.zip | |
[SPARK-17808][PYSPARK] Upgraded version of Pyrolite to 4.13
## What changes were proposed in this pull request?
Upgraded to a newer version of Pyrolite which supports serialization of a BinaryType StructField for PySpark.SQL
## How was this patch tested?
Added a unit test which fails with a raised ValueError when using the previous version of Pyrolite 4.9 and Python3
Author: Bryan Cutler <cutlerb@gmail.com>
Closes #15386 from BryanCutler/pyrolite-upgrade-SPARK-17808.
Diffstat (limited to 'python')
| -rw-r--r-- | python/pyspark/sql/tests.py | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/python/pyspark/sql/tests.py b/python/pyspark/sql/tests.py index 7b6f9f0ef1..86c590dae3 100644 --- a/python/pyspark/sql/tests.py +++ b/python/pyspark/sql/tests.py @@ -1708,6 +1708,14 @@ class SQLTests(ReusedPySparkTestCase): count = df.count() self.assertEquals(count, 4) + def test_BinaryType_serialization(self): + # Pyrolite version <= 4.9 could not serialize BinaryType with Python3 SPARK-17808 + schema = StructType([StructField('mybytes', BinaryType())]) + data = [[bytearray(b'here is my data')], + [bytearray(b'and here is some more')]] + df = self.spark.createDataFrame(data, schema=schema) + df.collect() + class HiveSparkSubmitTests(SparkSubmitTests): |