diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2015-05-20 16:42:49 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-05-20 16:42:49 -0700 |
| commit | f2faa7af30662e3bdf15780f8719c71108f8e30b (patch) | |
| tree | 0033bb03689beaf77a13f2e9062c21997f7a1e2f /unsafe/pom.xml | |
| parent | 7956dd7ab03e1542d89dd94c043f1e5131684199 (diff) | |
| download | spark-f2faa7af30662e3bdf15780f8719c71108f8e30b.tar.gz spark-f2faa7af30662e3bdf15780f8719c71108f8e30b.tar.bz2 spark-f2faa7af30662e3bdf15780f8719c71108f8e30b.zip | |
[SPARK-7251] Perform sequential scan when iterating over BytesToBytesMap
This patch modifies `BytesToBytesMap.iterator()` to iterate through records in the order that they appear in the data pages rather than iterating through the hashtable pointer arrays. This results in fewer random memory accesses, significantly improving performance for scan-and-copy operations.
This is possible because our data pages are laid out as sequences of `[keyLength][data][valueLength][data]` entries. In order to mark the end of a partially-filled data page, we write `-1` as a special end-of-page length (BytesToByesMap supports empty/zero-length keys and values, which is why we had to use a negative length).
This patch incorporates / closes #5836.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #6159 from JoshRosen/SPARK-7251 and squashes the following commits:
05bd90a [Josh Rosen] Compare capacity, not size, to MAX_CAPACITY
2a20d71 [Josh Rosen] Fix maximum BytesToBytesMap capacity
bc4854b [Josh Rosen] Guard against overflow when growing BytesToBytesMap
f5feadf [Josh Rosen] Add test for iterating over an empty map
273b842 [Josh Rosen] [SPARK-7251] Perform sequential scan when iterating over entries in BytesToBytesMap
Diffstat (limited to 'unsafe/pom.xml')
| -rw-r--r-- | unsafe/pom.xml | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/unsafe/pom.xml b/unsafe/pom.xml index 9e151fc7a9..2fd17267ac 100644 --- a/unsafe/pom.xml +++ b/unsafe/pom.xml @@ -65,6 +65,11 @@ <artifactId>junit-interface</artifactId> <scope>test</scope> </dependency> + <dependency> + <groupId>org.mockito</groupId> + <artifactId>mockito-all</artifactId> + <scope>test</scope> + </dependency> </dependencies> <build> <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory> |