diff options
| author | Chen Chao <crazyjvm@gmail.com> | 2014-05-14 18:20:20 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@apache.org> | 2014-05-14 18:20:20 -0700 |
| commit | 2f639957f0bf70dddf1e698aa9e26007fb58bc67 (patch) | |
| tree | d3992253e043746d7e1901ae19bf4c1433e911a6 /core | |
| parent | 44165fc91a31e6293a79031c89571e139d2c5356 (diff) | |
| download | spark-2f639957f0bf70dddf1e698aa9e26007fb58bc67.tar.gz spark-2f639957f0bf70dddf1e698aa9e26007fb58bc67.tar.bz2 spark-2f639957f0bf70dddf1e698aa9e26007fb58bc67.zip | |
default task number misleading in several places
private[streaming] def defaultPartitioner(numPartitions: Int = self.ssc.sc.defaultParallelism){
new HashPartitioner(numPartitions)
}
it represents that the default task number in Spark Streaming relies on the variable defaultParallelism in SparkContext, which is decided by the config property spark.default.parallelism
the property "spark.default.parallelism" refers to https://github.com/apache/spark/pull/389
Author: Chen Chao <crazyjvm@gmail.com>
Closes #766 from CrazyJvm/patch-7 and squashes the following commits:
0b7efba [Chen Chao] Update streaming-programming-guide.md
cc5b66c [Chen Chao] default task number misleading in several places
Diffstat (limited to 'core')
0 files changed, 0 insertions, 0 deletions