diff options
| author | Andrew Or <andrew@databricks.com> | 2015-12-01 19:51:12 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-12-01 19:51:12 -0800 |
| commit | d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6 (patch) | |
| tree | 18ce4722a49b29f9f22bf5d30d77d345c13f041c /R | |
| parent | 1ce4adf55b535518c2e63917a827fac1f2df4e8e (diff) | |
| download | spark-d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6.tar.gz spark-d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6.tar.bz2 spark-d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6.zip | |
[SPARK-12081] Make unified memory manager work with small heaps
The existing `spark.memory.fraction` (default 0.75) gives the system 25% of the space to work with. For small heaps, this is not enough: e.g. default 1GB leaves only 250MB system memory. This is especially a problem in local mode, where the driver and executor are crammed in the same JVM. Members of the community have reported driver OOM's in such cases.
**New proposal.** We now reserve 300MB before taking the 75%. For 1GB JVMs, this leaves `(1024 - 300) * 0.75 = 543MB` for execution and storage. This is proposal (1) listed in the [JIRA](https://issues.apache.org/jira/browse/SPARK-12081).
Author: Andrew Or <andrew@databricks.com>
Closes #10081 from andrewor14/unified-memory-small-heaps.
Diffstat (limited to 'R')
0 files changed, 0 insertions, 0 deletions