diff options
author | Kenichi Maehashi <webmaster@kenichimaehashi.com> | 2014-11-19 12:11:09 -0800 |
---|---|---|
committer | Andrew Or <andrew@databricks.com> | 2014-11-19 12:11:09 -0800 |
commit | eacc788346ccae232bd530dd880f801475a49734 (patch) | |
tree | b53cf3b0dc76ea7b57bd4b839e26eff8c6f180a9 /core/src | |
parent | d75579d09912cfb1eeac0589d625ea0452701fa0 (diff) | |
download | spark-eacc788346ccae232bd530dd880f801475a49734.tar.gz spark-eacc788346ccae232bd530dd880f801475a49734.tar.bz2 spark-eacc788346ccae232bd530dd880f801475a49734.zip |
[SPARK-4470] Validate number of threads in local mode
When running Spark locally, if number of threads is specified as 0 (e.g., `spark-submit --master local[0] ...`), the job got stuck and does not run at all.
I think it's better to validate the parameter.
Fix for [SPARK-4470](https://issues.apache.org/jira/browse/SPARK-4470).
Author: Kenichi Maehashi <webmaster@kenichimaehashi.com>
Closes #3337 from kmaehashi/spark-4470 and squashes the following commits:
3ad76f3 [Kenichi Maehashi] fix code style
7716734 [Kenichi Maehashi] SPARK-4470: Validate number of threads in local mode
Diffstat (limited to 'core/src')
-rw-r--r-- | core/src/main/scala/org/apache/spark/SparkContext.scala | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala index 37013121c5..ae8bbfb56f 100644 --- a/core/src/main/scala/org/apache/spark/SparkContext.scala +++ b/core/src/main/scala/org/apache/spark/SparkContext.scala @@ -1813,6 +1813,9 @@ object SparkContext extends Logging { def localCpuCount = Runtime.getRuntime.availableProcessors() // local[*] estimates the number of cores on the machine; local[N] uses exactly N threads. val threadCount = if (threads == "*") localCpuCount else threads.toInt + if (threadCount <= 0) { + throw new SparkException(s"Asked to run locally with $threadCount threads") + } val scheduler = new TaskSchedulerImpl(sc, MAX_LOCAL_TASK_FAILURES, isLocal = true) val backend = new LocalBackend(scheduler, threadCount) scheduler.initialize(backend) |