aboutsummaryrefslogtreecommitdiff
path: root/sql
diff options
context:
space:
mode:
authorAndrew Or <andrewor14@gmail.com>2014-06-20 17:14:33 -0700
committerReynold Xin <rxin@apache.org>2014-06-20 17:14:33 -0700
commit01125a1162b7bf42ae907cb6616616cc4ffb6fa9 (patch)
tree16d327946d0c37c1c8c9f122bd919e5a0c6d1296 /sql
parent0ac71d1284cd4f011d5763181cba9ecb49337b66 (diff)
downloadspark-01125a1162b7bf42ae907cb6616616cc4ffb6fa9.tar.gz
spark-01125a1162b7bf42ae907cb6616616cc4ffb6fa9.tar.bz2
spark-01125a1162b7bf42ae907cb6616616cc4ffb6fa9.zip
Clean up CacheManager et al.
**UPDATE** I have removed the special handling for `StorageLevel.MEMORY_*_SER` for now, because it introduces a potential performance regression. With the latest changes, this PR should include mainly style (code readability) fixes. The only functionality change is the update in `MemoryStore#putBytes` to actually return updated blocks, though this is a minor bug fix. Now this is mainly a precursor to another PR (once again). --------- *Old comment* The deserialized version of a partition may occupy much more space than the serialized version. Therefore, if a partition is to be cached with `StorageLevel.MEMORY_*_SER`, we don't need to fully unroll it into an `ArrayBuffer`, but instead we can unroll it into a potentially much smaller `ByteBuffer`. This may save us from OOMs in this case. Author: Andrew Or <andrewor14@gmail.com> Closes #1083 from andrewor14/unroll-them-partitions and squashes the following commits: 7048aa0 [Andrew Or] Merge branch 'master' of github.com:apache/spark into unroll-them-partitions 3d9a366 [Andrew Or] Minor change for readability d12b95f [Andrew Or] Remove unused imports (minor) a4c387b [Andrew Or] Merge branch 'master' of github.com:apache/spark into unroll-them-partitions cf5f565 [Andrew Or] Remove special handling for MEM_*_SER 0091ec0 [Andrew Or] Address review feedback 44ef282 [Andrew Or] Actually return updated blocks in putBytes 2941c89 [Andrew Or] Clean up BlockStore (minor) a8f181d [Andrew Or] Add special handling for StorageLevel.MEMORY_*_SER
Diffstat (limited to 'sql')
0 files changed, 0 insertions, 0 deletions