diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configuration.md | 4 | ||||
| -rw-r--r-- | docs/tuning.md | 2 |
2 files changed, 3 insertions, 3 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 741d6b2b37..c39b489085 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -719,8 +719,8 @@ Apart from these, the following properties are also available, and may be useful <td><code>spark.memory.fraction</code></td> <td>0.75</td> <td> - Fraction of the heap space used for execution and storage. The lower this is, the more - frequently spills and cached data eviction occur. The purpose of this config is to set + Fraction of (heap space - 300MB) used for execution and storage. The lower this is, the + more frequently spills and cached data eviction occur. The purpose of this config is to set aside memory for internal metadata, user data structures, and imprecise size estimation in the case of sparse, unusually large records. Leaving this at the default value is recommended. For more detail, see <a href="tuning.html#memory-management-overview"> diff --git a/docs/tuning.md b/docs/tuning.md index a8fe7a4532..e73ed69ffb 100644 --- a/docs/tuning.md +++ b/docs/tuning.md @@ -114,7 +114,7 @@ variety of workloads without requiring user expertise of how memory is divided i Although there are two relevant configurations, the typical user should not need to adjust them as the default values are applicable to most workloads: -* `spark.memory.fraction` expresses the size of `M` as a fraction of the total JVM heap space +* `spark.memory.fraction` expresses the size of `M` as a fraction of the (JVM heap space - 300MB) (default 0.75). The rest of the space (25%) is reserved for user data structures, internal metadata in Spark, and safeguarding against OOM errors in the case of sparse and unusually large records. |