diff options
author | Andrew Or <andrewor14@gmail.com> | 2014-06-20 17:14:33 -0700 |
---|---|---|
committer | Reynold Xin <rxin@apache.org> | 2014-06-20 17:14:33 -0700 |
commit | 01125a1162b7bf42ae907cb6616616cc4ffb6fa9 (patch) | |
tree | 16d327946d0c37c1c8c9f122bd919e5a0c6d1296 /sql | |
parent | 0ac71d1284cd4f011d5763181cba9ecb49337b66 (diff) | |
download | spark-01125a1162b7bf42ae907cb6616616cc4ffb6fa9.tar.gz spark-01125a1162b7bf42ae907cb6616616cc4ffb6fa9.tar.bz2 spark-01125a1162b7bf42ae907cb6616616cc4ffb6fa9.zip |
Clean up CacheManager et al.
**UPDATE**
I have removed the special handling for `StorageLevel.MEMORY_*_SER` for now, because it introduces a potential performance regression. With the latest changes, this PR should include mainly style (code readability) fixes. The only functionality change is the update in `MemoryStore#putBytes` to actually return updated blocks, though this is a minor bug fix.
Now this is mainly a precursor to another PR (once again).
---------
*Old comment*
The deserialized version of a partition may occupy much more space than the serialized version. Therefore, if a partition is to be cached with `StorageLevel.MEMORY_*_SER`, we don't need to fully unroll it into an `ArrayBuffer`, but instead we can unroll it into a potentially much smaller `ByteBuffer`. This may save us from OOMs in this case.
Author: Andrew Or <andrewor14@gmail.com>
Closes #1083 from andrewor14/unroll-them-partitions and squashes the following commits:
7048aa0 [Andrew Or] Merge branch 'master' of github.com:apache/spark into unroll-them-partitions
3d9a366 [Andrew Or] Minor change for readability
d12b95f [Andrew Or] Remove unused imports (minor)
a4c387b [Andrew Or] Merge branch 'master' of github.com:apache/spark into unroll-them-partitions
cf5f565 [Andrew Or] Remove special handling for MEM_*_SER
0091ec0 [Andrew Or] Address review feedback
44ef282 [Andrew Or] Actually return updated blocks in putBytes
2941c89 [Andrew Or] Clean up BlockStore (minor)
a8f181d [Andrew Or] Add special handling for StorageLevel.MEMORY_*_SER
Diffstat (limited to 'sql')
0 files changed, 0 insertions, 0 deletions