diff options
| author | Nishkam Ravi <nravi@cloudera.com> | 2014-10-02 13:48:35 -0500 |
|---|---|---|
| committer | Thomas Graves <tgraves@apache.org> | 2014-10-02 13:48:35 -0500 |
| commit | b4fb7b80a0d863500943d788ad3e34d502a6dafa (patch) | |
| tree | 2f2057289e9c37ae7579a01c4cdd65052bd348c4 /docs | |
| parent | 82a6a083a485140858bcd93d73adec59bb5cca64 (diff) | |
| download | spark-b4fb7b80a0d863500943d788ad3e34d502a6dafa.tar.gz spark-b4fb7b80a0d863500943d788ad3e34d502a6dafa.tar.bz2 spark-b4fb7b80a0d863500943d788ad3e34d502a6dafa.zip | |
Modify default YARN memory_overhead-- from an additive constant to a multiplier
Redone against the recent master branch (https://github.com/apache/spark/pull/1391)
Author: Nishkam Ravi <nravi@cloudera.com>
Author: nravi <nravi@c1704.halxg.cloudera.com>
Author: nishkamravi2 <nishkamravi@gmail.com>
Closes #2485 from nishkamravi2/master_nravi and squashes the following commits:
636a9ff [nishkamravi2] Update YarnAllocator.scala
8f76c8b [Nishkam Ravi] Doc change for yarn memory overhead
35daa64 [Nishkam Ravi] Slight change in the doc for yarn memory overhead
5ac2ec1 [Nishkam Ravi] Remove out
dac1047 [Nishkam Ravi] Additional documentation for yarn memory overhead issue
42c2c3d [Nishkam Ravi] Additional changes for yarn memory overhead issue
362da5e [Nishkam Ravi] Additional changes for yarn memory overhead
c726bd9 [Nishkam Ravi] Merge branch 'master' of https://github.com/apache/spark into master_nravi
f00fa31 [Nishkam Ravi] Improving logging for AM memoryOverhead
1cf2d1e [nishkamravi2] Update YarnAllocator.scala
ebcde10 [Nishkam Ravi] Modify default YARN memory_overhead-- from an additive constant to a multiplier (redone to resolve merge conflicts)
2e69f11 [Nishkam Ravi] Merge branch 'master' of https://github.com/apache/spark into master_nravi
efd688a [Nishkam Ravi] Merge branch 'master' of https://github.com/apache/spark
2b630f9 [nravi] Accept memory input as "30g", "512M" instead of an int value, to be consistent with rest of Spark
3bf8fad [nravi] Merge branch 'master' of https://github.com/apache/spark
5423a03 [nravi] Merge branch 'master' of https://github.com/apache/spark
eb663ca [nravi] Merge branch 'master' of https://github.com/apache/spark
df2aeb1 [nravi] Improved fix for ConcurrentModificationIssue (Spark-1097, Hadoop-10456)
6b840f0 [nravi] Undo the fix for SPARK-1758 (the problem is fixed)
5108700 [nravi] Fix in Spark for the Concurrent thread modification issue (SPARK-1097, HADOOP-10456)
681b36f [nravi] Fix for SPARK-1758: failing test org.apache.spark.JavaAPISuite.wholeTextFiles
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/running-on-yarn.md | 8 |
1 files changed, 4 insertions, 4 deletions
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 4b3a49eca7..695813a2ba 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -79,16 +79,16 @@ Most of the configs are the same for Spark on YARN as for other deployment modes </tr> <tr> <td><code>spark.yarn.executor.memoryOverhead</code></td> - <td>384</td> + <td>executorMemory * 0.07, with minimum of 384 </td> <td> - The amount of off heap memory (in megabytes) to be allocated per executor. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. + The amount of off heap memory (in megabytes) to be allocated per executor. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the executor size (typically 6-10%). </td> </tr> <tr> <td><code>spark.yarn.driver.memoryOverhead</code></td> - <td>384</td> + <td>driverMemory * 0.07, with minimum of 384 </td> <td> - The amount of off heap memory (in megabytes) to be allocated per driver. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. + The amount of off heap memory (in megabytes) to be allocated per driver. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the container size (typically 6-10%). </td> </tr> <tr> |