diff options
author | Josh Rosen <joshrosen@databricks.com> | 2015-10-25 21:19:52 -0700 |
---|---|---|
committer | Josh Rosen <joshrosen@databricks.com> | 2015-10-25 21:19:52 -0700 |
commit | 85e654c5ec87e666a8845bfd77185c1ea57b268a (patch) | |
tree | 2beadbc8fbb54369325970a4e2c7189506efad89 /tags | |
parent | 63accc79625d8a03d0624717af5e1d81b18a6da3 (diff) | |
download | spark-85e654c5ec87e666a8845bfd77185c1ea57b268a.tar.gz spark-85e654c5ec87e666a8845bfd77185c1ea57b268a.tar.bz2 spark-85e654c5ec87e666a8845bfd77185c1ea57b268a.zip |
[SPARK-10984] Simplify *MemoryManager class structure
This patch refactors the MemoryManager class structure. After #9000, Spark had the following classes:
- MemoryManager
- StaticMemoryManager
- ExecutorMemoryManager
- TaskMemoryManager
- ShuffleMemoryManager
This is fairly confusing. To simplify things, this patch consolidates several of these classes:
- ShuffleMemoryManager and ExecutorMemoryManager were merged into MemoryManager.
- TaskMemoryManager is moved into Spark Core.
**Key changes and tasks**:
- [x] Merge ExecutorMemoryManager into MemoryManager.
- [x] Move pooling logic into Allocator.
- [x] Move TaskMemoryManager from `spark-unsafe` to `spark-core`.
- [x] Refactor the existing Tungsten TaskMemoryManager interactions so Tungsten code use only this and not both this and ShuffleMemoryManager.
- [x] Refactor non-Tungsten code to use the TaskMemoryManager instead of ShuffleMemoryManager.
- [x] Merge ShuffleMemoryManager into MemoryManager.
- [x] Move code
- [x] ~~Simplify 1/n calculation.~~ **Will defer to followup, since this needs more work.**
- [x] Port ShuffleMemoryManagerSuite tests.
- [x] Move classes from `unsafe` package to `memory` package.
- [ ] Figure out how to handle the hacky use of the memory managers in HashedRelation's broadcast variable construction.
- [x] Test porting and cleanup: several tests relied on mock functionality (such as `TestShuffleMemoryManager.markAsOutOfMemory`) which has been changed or broken during the memory manager consolidation
- [x] AbstractBytesToBytesMapSuite
- [x] UnsafeExternalSorterSuite
- [x] UnsafeFixedWidthAggregationMapSuite
- [x] UnsafeKVExternalSorterSuite
**Compatiblity notes**:
- This patch introduces breaking changes in `ExternalAppendOnlyMap`, which is marked as `DevloperAPI` (likely for legacy reasons): this class now cannot be used outside of a task.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #9127 from JoshRosen/SPARK-10984.
Diffstat (limited to 'tags')
0 files changed, 0 insertions, 0 deletions