diff options
| author | Davies Liu <davies@databricks.com> | 2016-09-06 10:46:31 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-09-06 10:46:31 -0700 |
| commit | f7e26d788757f917b32749856bb29feb7b4c2987 (patch) | |
| tree | b3efe1ff2f73e345c1e4c955239c648e0b9739ca /pom.xml | |
| parent | bc2767df2666ff615e7f44e980555afab06dd8a3 (diff) | |
| download | spark-f7e26d788757f917b32749856bb29feb7b4c2987.tar.gz spark-f7e26d788757f917b32749856bb29feb7b4c2987.tar.bz2 spark-f7e26d788757f917b32749856bb29feb7b4c2987.zip | |
[SPARK-16922] [SPARK-17211] [SQL] make the address of values portable in LongToUnsafeRowMap
## What changes were proposed in this pull request?
In LongToUnsafeRowMap, we use offset of a value as pointer, stored in a array also in the page for chained values. The offset is not portable, because Platform.LONG_ARRAY_OFFSET will be different with different JVM Heap size, then the deserialized LongToUnsafeRowMap will be corrupt.
This PR will change to use portable address (without Platform.LONG_ARRAY_OFFSET).
## How was this patch tested?
Added a test case with random generated keys, to improve the coverage. But this test is not a regression test, that could require a Spark cluster that have at least 32G heap in driver or executor.
Author: Davies Liu <davies@databricks.com>
Closes #14927 from davies/longmap.
Diffstat (limited to 'pom.xml')
0 files changed, 0 insertions, 0 deletions