diff options
| author | Aaron Davidson <aaron@databricks.com> | 2014-11-25 23:57:04 -0500 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-11-25 23:57:34 -0500 |
| commit | 1e12f594be277f6b390c998b1a1e5581ecebdcb0 (patch) | |
| tree | 8874150e0e5c8dde1e1bb814493012eb7f4e439a /yarn | |
| parent | 2756d0de91d996f80c0b0883cad1d2fab336ed84 (diff) | |
| download | spark-1e12f594be277f6b390c998b1a1e5581ecebdcb0.tar.gz spark-1e12f594be277f6b390c998b1a1e5581ecebdcb0.tar.bz2 spark-1e12f594be277f6b390c998b1a1e5581ecebdcb0.zip | |
[SPARK-4516] Cap default number of Netty threads at 8
In practice, only 2-4 cores should be required to transfer roughly 10 Gb/s, and each core that we use will have an initial overhead of roughly 32 MB of off-heap memory, which comes at a premium.
Thus, this value should still retain maximum throughput and reduce wasted off-heap memory allocation. It can be overridden by setting the number of serverThreads and clientThreads manually in Spark's configuration.
Author: Aaron Davidson <aaron@databricks.com>
Closes #3469 from aarondav/fewer-pools2 and squashes the following commits:
087c59f [Aaron Davidson] [SPARK-4516] Cap default number of Netty threads at 8
(cherry picked from commit f5f2d27385c243959f03a9d78a149d5f405b2f50)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Diffstat (limited to 'yarn')
0 files changed, 0 insertions, 0 deletions