diff options
| author | Sean Owen <sowen@cloudera.com> | 2016-06-16 23:04:10 +0200 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-06-16 23:04:10 +0200 |
| commit | 457126e420e66228cc68def4bc3d87e7a282069a (patch) | |
| tree | 70a86161312c672cd7d85cedcd3aaa281d49c1f7 /docs/configuration.md | |
| parent | 36110a8306608186696c536028d2776e022d305a (diff) | |
| download | spark-457126e420e66228cc68def4bc3d87e7a282069a.tar.gz spark-457126e420e66228cc68def4bc3d87e7a282069a.tar.bz2 spark-457126e420e66228cc68def4bc3d87e7a282069a.zip | |
[SPARK-15796][CORE] Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config
## What changes were proposed in this pull request?
Reduce `spark.memory.fraction` default to 0.6 in order to make it fit within default JVM old generation size (2/3 heap). See JIRA discussion. This means a full cache doesn't spill into the new gen. CC andrewor14
## How was this patch tested?
Jenkins tests.
Author: Sean Owen <sowen@cloudera.com>
Closes #13618 from srowen/SPARK-15796.
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 7 |
1 files changed, 4 insertions, 3 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 32c3a92660..fbda91c109 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -778,14 +778,15 @@ Apart from these, the following properties are also available, and may be useful <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr> <tr> <td><code>spark.memory.fraction</code></td> - <td>0.75</td> + <td>0.6</td> <td> Fraction of (heap space - 300MB) used for execution and storage. The lower this is, the more frequently spills and cached data eviction occur. The purpose of this config is to set aside memory for internal metadata, user data structures, and imprecise size estimation in the case of sparse, unusually large records. Leaving this at the default value is - recommended. For more detail, see <a href="tuning.html#memory-management-overview"> - this description</a>. + recommended. For more detail, including important information about correctly tuning JVM + garbage collection when increasing this value, see + <a href="tuning.html#memory-management-overview">this description</a>. </td> </tr> <tr> |