diff options
| author | Chen Chao <crazyjvm@gmail.com> | 2014-03-03 14:41:25 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@apache.org> | 2014-03-03 14:41:25 -0800 |
| commit | 9d225a91043ac92a0e727ba281b10c250a945614 (patch) | |
| tree | d3f5016a6de75bc05763561517ee91d549908725 /docs/tuning.md | |
| parent | 369aad6f9e25d49b0d32f37e5069b6d4e41a59d9 (diff) | |
| download | spark-9d225a91043ac92a0e727ba281b10c250a945614.tar.gz spark-9d225a91043ac92a0e727ba281b10c250a945614.tar.bz2 spark-9d225a91043ac92a0e727ba281b10c250a945614.zip | |
update proportion of memory
The default value of "spark.storage.memoryFraction" has been changed from 0.66 to 0.6 . So it should be 60% of the memory to cache while 40% used for task execution.
Author: Chen Chao <crazyjvm@gmail.com>
Closes #66 from CrazyJvm/master and squashes the following commits:
0f84d86 [Chen Chao] update proportion of memory
Diffstat (limited to 'docs/tuning.md')
| -rw-r--r-- | docs/tuning.md | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/docs/tuning.md b/docs/tuning.md index 704778681c..26ff1325bb 100644 --- a/docs/tuning.md +++ b/docs/tuning.md @@ -163,8 +163,8 @@ their work directories), *not* on your driver program. **Cache Size Tuning** One important configuration parameter for GC is the amount of memory that should be used for caching RDDs. -By default, Spark uses 66% of the configured executor memory (`spark.executor.memory` or `SPARK_MEM`) to -cache RDDs. This means that 33% of memory is available for any objects created during task execution. +By default, Spark uses 60% of the configured executor memory (`spark.executor.memory` or `SPARK_MEM`) to +cache RDDs. This means that 40% of memory is available for any objects created during task execution. In case your tasks slow down and you find that your JVM is garbage-collecting frequently or running out of memory, lowering this value will help reduce the memory consumption. To change this to say 50%, you can call |