diff options
| author | Andrew Or <andrew@databricks.com> | 2015-12-01 19:51:12 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-12-01 19:51:12 -0800 |
| commit | d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6 (patch) | |
| tree | 18ce4722a49b29f9f22bf5d30d77d345c13f041c /docs/configuration.md | |
| parent | 1ce4adf55b535518c2e63917a827fac1f2df4e8e (diff) | |
| download | spark-d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6.tar.gz spark-d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6.tar.bz2 spark-d96f8c997b9bb5c3d61f513d2c71d67ccf8e85d6.zip | |
[SPARK-12081] Make unified memory manager work with small heaps
The existing `spark.memory.fraction` (default 0.75) gives the system 25% of the space to work with. For small heaps, this is not enough: e.g. default 1GB leaves only 250MB system memory. This is especially a problem in local mode, where the driver and executor are crammed in the same JVM. Members of the community have reported driver OOM's in such cases.
**New proposal.** We now reserve 300MB before taking the 75%. For 1GB JVMs, this leaves `(1024 - 300) * 0.75 = 543MB` for execution and storage. This is proposal (1) listed in the [JIRA](https://issues.apache.org/jira/browse/SPARK-12081).
Author: Andrew Or <andrew@databricks.com>
Closes #10081 from andrewor14/unified-memory-small-heaps.
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 741d6b2b37..c39b489085 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -719,8 +719,8 @@ Apart from these, the following properties are also available, and may be useful <td><code>spark.memory.fraction</code></td> <td>0.75</td> <td> - Fraction of the heap space used for execution and storage. The lower this is, the more - frequently spills and cached data eviction occur. The purpose of this config is to set + Fraction of (heap space - 300MB) used for execution and storage. The lower this is, the + more frequently spills and cached data eviction occur. The purpose of this config is to set aside memory for internal metadata, user data structures, and imprecise size estimation in the case of sparse, unusually large records. Leaving this at the default value is recommended. For more detail, see <a href="tuning.html#memory-management-overview"> |