diff options
| author | Fei Wang <wangfei1@huawei.com> | 2015-12-08 21:32:31 -0800 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2015-12-08 21:32:31 -0800 |
| commit | 3934562d34bbe08d91c54b4bbee27870e93d7571 (patch) | |
| tree | 44bbd6f2148f7127aecbec993493577384c36b92 /python | |
| parent | a0046e379bee0852c39ece4ea719cde70d350b0e (diff) | |
| download | spark-3934562d34bbe08d91c54b4bbee27870e93d7571.tar.gz spark-3934562d34bbe08d91c54b4bbee27870e93d7571.tar.bz2 spark-3934562d34bbe08d91c54b4bbee27870e93d7571.zip | |
[SPARK-12222] [CORE] Deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception
Jira: https://issues.apache.org/jira/browse/SPARK-12222
Deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception:
```
com.esotericsoftware.kryo.KryoException: Buffer underflow.
at com.esotericsoftware.kryo.io.Input.require(Input.java:156)
at com.esotericsoftware.kryo.io.Input.skip(Input.java:131)
at com.esotericsoftware.kryo.io.Input.skip(Input.java:264)
```
This is caused by a bug of kryo's `Input.skip(long count)`(https://github.com/EsotericSoftware/kryo/issues/119) and we call this method in `KryoInputDataInputBridge`.
Instead of upgrade kryo's version, this pr bypass the kryo's `Input.skip(long count)` by directly call another `skip` method in kryo's Input.java(https://github.com/EsotericSoftware/kryo/blob/kryo-2.21/src/com/esotericsoftware/kryo/io/Input.java#L124), i.e. write the bug-fixed version of `Input.skip(long count)` in KryoInputDataInputBridge's `skipBytes` method.
more detail link to https://github.com/apache/spark/pull/9748#issuecomment-162860246
Author: Fei Wang <wangfei1@huawei.com>
Closes #10213 from scwf/patch-1.
Diffstat (limited to 'python')
0 files changed, 0 insertions, 0 deletions