diff options
| author | Kousuke Saruta <sarutak@oss.nttdata.co.jp> | 2015-02-06 09:39:12 +0000 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2015-02-06 09:39:12 +0000 |
| commit | 24dbc50b934bfcd9fc3fd91cfd9966b75b250ad7 (patch) | |
| tree | 6fedc5c910679215d71fdebf55f198fe642d3547 /yarn | |
| parent | f6ba813af21f8aa277155b789a7330e356ce15da (diff) | |
| download | spark-24dbc50b934bfcd9fc3fd91cfd9966b75b250ad7.tar.gz spark-24dbc50b934bfcd9fc3fd91cfd9966b75b250ad7.tar.bz2 spark-24dbc50b934bfcd9fc3fd91cfd9966b75b250ad7.zip | |
[SPARK-5157][YARN] Configure more JVM options properly when we use ConcMarkSweepGC for AM.
When we set `SPARK_USE_CONC_INCR_GC`, ConcurrentMarkSweepGC works on the AM.
Actually, if ConcurrentMarkSweepGC is set for the JVM, following JVM options are set automatically and implicitly.
* MaxTenuringThreshold=0
* SurvivorRatio=1024
Those can not be proper value for most cases.
See also http://www.oracle.com/technetwork/java/tuning-139912.html
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Closes #3956 from sarutak/SPARK-5157 and squashes the following commits:
c15da4e [Kousuke Saruta] Set more JVM options for AM when enabling CMS
Diffstat (limited to 'yarn')
| -rw-r--r-- | yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala index 91e8574e94..e7005094b5 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala @@ -418,6 +418,8 @@ private[spark] class Client( // In our expts, using (default) throughput collector has severe perf ramifications in // multi-tenant machines javaOpts += "-XX:+UseConcMarkSweepGC" + javaOpts += "-XX:MaxTenuringThreshold=31" + javaOpts += "-XX:SurvivorRatio=8" javaOpts += "-XX:+CMSIncrementalMode" javaOpts += "-XX:+CMSIncrementalPacing" javaOpts += "-XX:CMSIncrementalDutyCycleMin=0" |