diff options
author | lisurprise <zhichao.li@intel.com> | 2015-03-16 13:10:32 -0700 |
---|---|---|
committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-03-16 13:10:32 -0700 |
commit | f149b8b5e542af44650923d0156f037121b45a20 (patch) | |
tree | 80fdef3c31ad323a5d0ff3d6e9827c78045413b4 /streaming/pom.xml | |
parent | d19efeddc0cb710c9496af11e447d39e1ad61b31 (diff) | |
download | spark-f149b8b5e542af44650923d0156f037121b45a20.tar.gz spark-f149b8b5e542af44650923d0156f037121b45a20.tar.bz2 spark-f149b8b5e542af44650923d0156f037121b45a20.zip |
[SPARK-6077] Remove streaming tab while stopping StreamingContext
Currently we would create a new streaming tab for each streamingContext even if there's already one on the same sparkContext which would cause duplicate StreamingTab created and none of them is taking effect.
snapshot: https://www.dropbox.com/s/t4gd6hqyqo0nivz/bad%20multiple%20streamings.png?dl=0
How to reproduce:
1)
import org.apache.spark.SparkConf
import org.apache.spark.streaming.
{Seconds, StreamingContext}
import org.apache.spark.storage.StorageLevel
val ssc = new StreamingContext(sc, Seconds(1))
val lines = ssc.socketTextStream("localhost", 9999, StorageLevel.MEMORY_AND_DISK_SER)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
wordCounts.print()
ssc.start()
.....
2)
ssc.stop(false)
val ssc = new StreamingContext(sc, Seconds(1))
val lines = ssc.socketTextStream("localhost", 9999, StorageLevel.MEMORY_AND_DISK_SER)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
wordCounts.print()
ssc.start()
Author: lisurprise <zhichao.li@intel.com>
Closes #4828 from zhichao-li/master and squashes the following commits:
c329806 [lisurprise] add test for attaching/detaching streaming tab
51e6c7f [lisurprise] move detach method into StreamingTab
31a44fa [lisurprise] add unit test for attaching and detaching new tab
db25ed2 [lisurprise] clean code
8281bcb [lisurprise] clean code
193c542 [lisurprise] remove streaming tab while closing streaming context
Diffstat (limited to 'streaming/pom.xml')
-rw-r--r-- | streaming/pom.xml | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/streaming/pom.xml b/streaming/pom.xml index 0370b0e9e1..96508d83f4 100644 --- a/streaming/pom.xml +++ b/streaming/pom.xml @@ -83,6 +83,11 @@ <scope>test</scope> </dependency> <dependency> + <groupId>org.seleniumhq.selenium</groupId> + <artifactId>selenium-java</artifactId> + <scope>test</scope> + </dependency> + <dependency> <groupId>com.novocode</groupId> <artifactId>junit-interface</artifactId> <scope>test</scope> |