diff options
| author | hushan[胡珊] <hushan@xiaomi.com> | 2015-03-02 16:53:54 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-03-02 16:53:54 -0800 |
| commit | e3a88d1104ebdb858f0509f56d7bb536037e5f63 (patch) | |
| tree | 849c18c12fc41e7284432c6cc89bee5e16061807 /python/pyspark/mllib/linalg.py | |
| parent | 258d154c9f1afdd52dce19f03d81683ee34effac (diff) | |
| download | spark-e3a88d1104ebdb858f0509f56d7bb536037e5f63.tar.gz spark-e3a88d1104ebdb858f0509f56d7bb536037e5f63.tar.bz2 spark-e3a88d1104ebdb858f0509f56d7bb536037e5f63.zip | |
[SPARK-4777][CORE] Some block memory after unrollSafely not count into used memory(memoryStore.entrys or unrollMemory)
Some memory not count into memory used by memoryStore or unrollMemory.
Thread A after unrollsafely memory, it will release 40MB unrollMemory(40MB will used by other threads). then ThreadA wait get accountingLock to tryToPut blockA(30MB). before Thread A get accountingLock, blockA memory size is not counting into unrollMemory or memoryStore.currentMemory.
IIUC, freeMemory should minus that block memory
So, put this release memory into pending, and release it in tryToPut before ensureSpace
Author: hushan[胡珊] <hushan@xiaomi.com>
Closes #3629 from suyanNone/unroll-memory and squashes the following commits:
809cc41 [hushan[胡珊]] Refine
407b2c9 [hushan[胡珊]] Refine according comments
39960d0 [hushan[胡珊]] Refine comments
0fd0213 [hushan[胡珊]] add comments
0fc2bec [hushan[胡珊]] Release pending unroll memory after put block in memoryStore
3a3f2c8 [hushan[胡珊]] Refine blockManagerSuite unroll test
3323c45 [hushan[胡珊]] Refine getOrElse
f664317 [hushan[胡珊]] Make sure not add pending in every releaseUnrollMemory call
08b32ba [hushan[胡珊]] Pending unroll memory for this block untill tryToPut
Diffstat (limited to 'python/pyspark/mllib/linalg.py')
0 files changed, 0 insertions, 0 deletions