aboutsummaryrefslogtreecommitdiff
path: root/docs/configuration.md
diff options
context:
space:
mode:
authorChen Chao <crazyjvm@gmail.com>2014-04-16 09:14:18 -0700
committerPatrick Wendell <pwendell@gmail.com>2014-04-16 09:14:18 -0700
commit9edd88782e0268439c5ab57400d6a7ab432fc269 (patch)
tree1a5c8b05925aec2e3f7f6953b54cc49ee9bfae05 /docs/configuration.md
parentfec462c15321fa138d9654d64cc08468b6ab5ad1 (diff)
downloadspark-9edd88782e0268439c5ab57400d6a7ab432fc269.tar.gz
spark-9edd88782e0268439c5ab57400d6a7ab432fc269.tar.bz2
spark-9edd88782e0268439c5ab57400d6a7ab432fc269.zip
update spark.default.parallelism
actually, the value 8 is only valid in mesos fine-grained mode : <code> override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8) </code> while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers! <code> override def defaultParallelism(): Int = { conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2)) } </code> Author: Chen Chao <crazyjvm@gmail.com> Closes #389 from CrazyJvm/patch-2 and squashes the following commits: 84a7fe4 [Chen Chao] miss </li> at the end of every single line 04a9796 [Chen Chao] change format ee0fae0 [Chen Chao] update spark.default.parallelism
Diffstat (limited to 'docs/configuration.md')
-rw-r--r--docs/configuration.md8
1 files changed, 7 insertions, 1 deletions
diff --git a/docs/configuration.md b/docs/configuration.md
index f3bfd036f4..a3029837ff 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -96,7 +96,13 @@ Apart from these, the following properties are also available, and may be useful
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr>
<td>spark.default.parallelism</td>
- <td>8</td>
+ <td>
+ <ul>
+ <li>Mesos fine grained mode: 8</li>
+ <li>Local mode: core number of the local machine</li>
+ <li>Others: total core number of all executor nodes or 2, whichever is larger</li>
+ </ul>
+ </td>
<td>
Default number of tasks to use across the cluster for distributed shuffle operations (<code>groupByKey</code>,
<code>reduceByKey</code>, etc) when not set by user.