diff options
author | lisurprise <zhichao.li@intel.com> | 2015-03-16 13:10:32 -0700 |
---|---|---|
committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-03-16 13:10:32 -0700 |
commit | f149b8b5e542af44650923d0156f037121b45a20 (patch) | |
tree | 80fdef3c31ad323a5d0ff3d6e9827c78045413b4 /python/pyspark | |
parent | d19efeddc0cb710c9496af11e447d39e1ad61b31 (diff) | |
download | spark-f149b8b5e542af44650923d0156f037121b45a20.tar.gz spark-f149b8b5e542af44650923d0156f037121b45a20.tar.bz2 spark-f149b8b5e542af44650923d0156f037121b45a20.zip |
[SPARK-6077] Remove streaming tab while stopping StreamingContext
Currently we would create a new streaming tab for each streamingContext even if there's already one on the same sparkContext which would cause duplicate StreamingTab created and none of them is taking effect.
snapshot: https://www.dropbox.com/s/t4gd6hqyqo0nivz/bad%20multiple%20streamings.png?dl=0
How to reproduce:
1)
import org.apache.spark.SparkConf
import org.apache.spark.streaming.
{Seconds, StreamingContext}
import org.apache.spark.storage.StorageLevel
val ssc = new StreamingContext(sc, Seconds(1))
val lines = ssc.socketTextStream("localhost", 9999, StorageLevel.MEMORY_AND_DISK_SER)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
wordCounts.print()
ssc.start()
.....
2)
ssc.stop(false)
val ssc = new StreamingContext(sc, Seconds(1))
val lines = ssc.socketTextStream("localhost", 9999, StorageLevel.MEMORY_AND_DISK_SER)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
wordCounts.print()
ssc.start()
Author: lisurprise <zhichao.li@intel.com>
Closes #4828 from zhichao-li/master and squashes the following commits:
c329806 [lisurprise] add test for attaching/detaching streaming tab
51e6c7f [lisurprise] move detach method into StreamingTab
31a44fa [lisurprise] add unit test for attaching and detaching new tab
db25ed2 [lisurprise] clean code
8281bcb [lisurprise] clean code
193c542 [lisurprise] remove streaming tab while closing streaming context
Diffstat (limited to 'python/pyspark')
0 files changed, 0 insertions, 0 deletions