diff options
| author | Andrew Or <andrew@databricks.com> | 2015-12-01 19:51:12 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-12-01 19:51:12 -0800 |
| commit | d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6 (patch) | |
| tree | 18ce4722a49b29f9f22bf5d30d77d345c13f041c /docs/tuning.md | |
| parent | 1ce4adf55b535518c2e63917a827fac1f2df4e8e (diff) | |
| download | spark-d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6.tar.gz spark-d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6.tar.bz2 spark-d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6.zip | |
[SPARK-12081] Make unified memory manager work with small heaps
The existing `spark.memory.fraction` (default 0.75) gives the system 25% of the space to work with. For small heaps, this is not enough: e.g. default 1GB leaves only 250MB system memory. This is especially a problem in local mode, where the driver and executor are crammed in the same JVM. Members of the community have reported driver OOM's in such cases.
**New proposal.** We now reserve 300MB before taking the 75%. For 1GB JVMs, this leaves `(1024 - 300) * 0.75 = 543MB` for execution and storage. This is proposal (1) listed in the [JIRA](https://issues.apache.org/jira/browse/SPARK-12081).
Author: Andrew Or <andrew@databricks.com>
Closes #10081 from andrewor14/unified-memory-small-heaps.
Diffstat (limited to 'docs/tuning.md')
| -rw-r--r-- | docs/tuning.md | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/docs/tuning.md b/docs/tuning.md index a8fe7a4532..e73ed69ffb 100644 --- a/docs/tuning.md +++ b/docs/tuning.md @@ -114,7 +114,7 @@ variety of workloads without requiring user expertise of how memory is divided i Although there are two relevant configurations, the typical user should not need to adjust them as the default values are applicable to most workloads: -* `spark.memory.fraction` expresses the size of `M` as a fraction of the total JVM heap space +* `spark.memory.fraction` expresses the size of `M` as a fraction of the (JVM heap space - 300MB) (default 0.75). The rest of the space (25%) is reserved for user data structures, internal metadata in Spark, and safeguarding against OOM errors in the case of sparse and unusually large records. |