diff options
author | Josh Rosen <joshrosen@databricks.com> | 2016-03-14 14:26:39 -0700 |
---|---|---|
committer | Andrew Or <andrew@databricks.com> | 2016-03-14 14:26:39 -0700 |
commit | 38529d8f2350feb1f143aab0be336050c0f887f2 (patch) | |
tree | e5d24f4b426432b7b9cdc7085685597c7b5921b8 /external | |
parent | a48296f4fe513b63041f1a26231cfe152b69657f (diff) | |
download | spark-38529d8f2350feb1f143aab0be336050c0f887f2.tar.gz spark-38529d8f2350feb1f143aab0be336050c0f887f2.tar.bz2 spark-38529d8f2350feb1f143aab0be336050c0f887f2.zip |
[SPARK-10907][SPARK-6157] Remove pendingUnrollMemory from MemoryStore
This patch refactors the MemoryStore to remove the concept of `pendingUnrollMemory`. It also fixes fixes SPARK-6157: "Unrolling with MEMORY_AND_DISK should always release memory".
Key changes:
- Inline `MemoryStore.tryToPut` at its three call sites in the `MemoryStore`.
- Inline `Memory.unrollSafely` at its only call site (in `MemoryStore.putIterator`).
- Inline `MemoryManager.acquireStorageMemory` at its call sites.
- Simplify the code as a result of this inlining (some parameters have fixed values after inlining, so lots of branches can be removed).
- Remove the `pendingUnrollMemory` map by returning the amount of unrollMemory allocated when returning an iterator after a failed `putIterator` call.
- Change `putIterator` to return an instance of `PartiallyUnrolledIterator`, a special iterator subclass which will automatically free the unroll memory of its partially-unrolled elements when the iterator is consumed. To handle cases where the iterator is not consumed (e.g. when a MEMORY_ONLY put fails), `PartiallyUnrolledIterator` exposes a `close()` method which may be called to discard the unrolled values and free their memory.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #11613 from JoshRosen/cleanup-unroll-memory.
Diffstat (limited to 'external')
0 files changed, 0 insertions, 0 deletions