diff options
| author | jerryshao <saisai.shao@intel.com> | 2014-11-25 05:36:29 -0800 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2014-11-25 05:36:29 -0800 |
| commit | fef27b29431c2adadc17580f26c23afa6a3bd1d2 (patch) | |
| tree | d8c58485957d4d3ea312c1b1a0be424c986cc480 /streaming | |
| parent | a51118a34a4617c07373480c4b021e53124c3c00 (diff) | |
| download | spark-fef27b29431c2adadc17580f26c23afa6a3bd1d2.tar.gz spark-fef27b29431c2adadc17580f26c23afa6a3bd1d2.tar.bz2 spark-fef27b29431c2adadc17580f26c23afa6a3bd1d2.zip | |
[SPARK-4381][Streaming]Add warning log when user set spark.master to local in Spark Streaming and there's no job executed
Author: jerryshao <saisai.shao@intel.com>
Closes #3244 from jerryshao/SPARK-4381 and squashes the following commits:
d2486c7 [jerryshao] Improve the warning log
d726e85 [jerryshao] Add local[1] to the filter condition
eca428b [jerryshao] Add warning log
Diffstat (limited to 'streaming')
| -rw-r--r-- | streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala b/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala index ec59221459..ecab5510a8 100644 --- a/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala +++ b/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala @@ -121,6 +121,11 @@ class StreamingContext private[streaming] ( } } + if (sc.conf.get("spark.master") == "local" || sc.conf.get("spark.master") == "local[1]") { + logWarning("spark.master should be set as local[n], n > 1 in local mode if you have receivers" + + " to get data, otherwise Spark jobs will not get resources to process the received data.") + } + private[streaming] val conf = sc.conf private[streaming] val env = SparkEnv.get |