diff options
| author | Matei Zaharia <matei@eecs.berkeley.edu> | 2013-10-29 23:47:10 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@eecs.berkeley.edu> | 2013-10-29 23:47:10 -0700 |
| commit | 745dc4290834e8c38d61600e23e4706b0ae47179 (patch) | |
| tree | 8002b4340378b8561a043d9e10e55561a980f4dc /pom.xml | |
| parent | f0e23a023ce1356bc0f04248605c48d4d08c2d05 (diff) | |
| parent | cb9c8a922fc92f55e1de87ff13cbdbc3990c4ee3 (diff) | |
| download | spark-745dc4290834e8c38d61600e23e4706b0ae47179.tar.gz spark-745dc4290834e8c38d61600e23e4706b0ae47179.tar.bz2 spark-745dc4290834e8c38d61600e23e4706b0ae47179.zip | |
Merge pull request #118 from JoshRosen/blockinfo-memory-usage
Reduce the memory footprint of BlockInfo objects
This pull request reduces the memory footprint of all BlockInfo objects and makes additional optimizations for shuffle blocks. For all BlockInfo objects, these changes remove two boolean fields and one Object field. For shuffle blocks, we additionally remove an Object field and a boolean field.
When storing tens of thousands of these objects, this may add up to significant memory savings. A ShuffleBlockInfo now only needs to wrap a single long.
This was motivated by a [report of high blockInfo memory usage during shuffles](https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201310.mbox/%3C20131026134353.202b2b9b%40sh9%3E).
I haven't run benchmarks to measure the exact memory savings.
/cc @aarondav
Diffstat (limited to 'pom.xml')
0 files changed, 0 insertions, 0 deletions