aboutsummaryrefslogtreecommitdiff
path: root/core
diff options
context:
space:
mode:
authorKenichi Maehashi <webmaster@kenichimaehashi.com>2014-11-19 12:11:09 -0800
committerAndrew Or <andrew@databricks.com>2014-11-19 12:11:09 -0800
commiteacc788346ccae232bd530dd880f801475a49734 (patch)
treeb53cf3b0dc76ea7b57bd4b839e26eff8c6f180a9 /core
parentd75579d09912cfb1eeac0589d625ea0452701fa0 (diff)
downloadspark-eacc788346ccae232bd530dd880f801475a49734.tar.gz
spark-eacc788346ccae232bd530dd880f801475a49734.tar.bz2
spark-eacc788346ccae232bd530dd880f801475a49734.zip
[SPARK-4470] Validate number of threads in local mode
When running Spark locally, if number of threads is specified as 0 (e.g., `spark-submit --master local[0] ...`), the job got stuck and does not run at all. I think it's better to validate the parameter. Fix for [SPARK-4470](https://issues.apache.org/jira/browse/SPARK-4470). Author: Kenichi Maehashi <webmaster@kenichimaehashi.com> Closes #3337 from kmaehashi/spark-4470 and squashes the following commits: 3ad76f3 [Kenichi Maehashi] fix code style 7716734 [Kenichi Maehashi] SPARK-4470: Validate number of threads in local mode
Diffstat (limited to 'core')
-rw-r--r--core/src/main/scala/org/apache/spark/SparkContext.scala3
1 files changed, 3 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 37013121c5..ae8bbfb56f 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -1813,6 +1813,9 @@ object SparkContext extends Logging {
def localCpuCount = Runtime.getRuntime.availableProcessors()
// local[*] estimates the number of cores on the machine; local[N] uses exactly N threads.
val threadCount = if (threads == "*") localCpuCount else threads.toInt
+ if (threadCount <= 0) {
+ throw new SparkException(s"Asked to run locally with $threadCount threads")
+ }
val scheduler = new TaskSchedulerImpl(sc, MAX_LOCAL_TASK_FAILURES, isLocal = true)
val backend = new LocalBackend(scheduler, threadCount)
scheduler.initialize(backend)