diff options
author | Patrick Wendell <pwendell@gmail.com> | 2014-01-14 00:05:37 -0800 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-01-14 00:05:37 -0800 |
commit | 980250b1ee0cdba9cf06ea87c790a2d504bbf03e (patch) | |
tree | 547a023f4a672ab84299762d71e40d19dc2cf92f /examples | |
parent | 055be5c6940a82e2d7fa39f968a434643a1eb1e2 (diff) | |
parent | f8bd828c7ccf1ff69bc35bf95d07183cb35a7c72 (diff) | |
download | spark-980250b1ee0cdba9cf06ea87c790a2d504bbf03e.tar.gz spark-980250b1ee0cdba9cf06ea87c790a2d504bbf03e.tar.bz2 spark-980250b1ee0cdba9cf06ea87c790a2d504bbf03e.zip |
Merge pull request #416 from tdas/filestream-fix
Removed unnecessary DStream operations and updated docs
Removed StreamingContext.registerInputStream and registerOutputStream - they were useless. InputDStream has been made to register itself, and just registering a DStream as output stream cause RDD objects to be created but the RDDs will not be computed at all.. Also made DStream.register() private[streaming] for the same reasons.
Updated docs, specially added package documentation for streaming package.
Also, changed NetworkWordCount's input storage level to use MEMORY_ONLY, replication on the local machine causes warning messages (as replication fails) which is scary for a new user trying out his/her first example.
Diffstat (limited to 'examples')
-rw-r--r-- | examples/src/main/scala/org/apache/spark/streaming/examples/NetworkWordCount.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/examples/src/main/scala/org/apache/spark/streaming/examples/NetworkWordCount.scala b/examples/src/main/scala/org/apache/spark/streaming/examples/NetworkWordCount.scala index 25f7013307..0226475712 100644 --- a/examples/src/main/scala/org/apache/spark/streaming/examples/NetworkWordCount.scala +++ b/examples/src/main/scala/org/apache/spark/streaming/examples/NetworkWordCount.scala @@ -19,6 +19,7 @@ package org.apache.spark.streaming.examples import org.apache.spark.streaming.{Seconds, StreamingContext} import org.apache.spark.streaming.StreamingContext._ +import org.apache.spark.storage.StorageLevel /** * Counts words in text encoded with UTF8 received from the network every second. @@ -48,7 +49,7 @@ object NetworkWordCount { // Create a NetworkInputDStream on target ip:port and count the // words in input stream of \n delimited text (eg. generated by 'nc') - val lines = ssc.socketTextStream(args(1), args(2).toInt) + val lines = ssc.socketTextStream(args(1), args(2).toInt, StorageLevel.MEMORY_ONLY_SER) val words = lines.flatMap(_.split(" ")) val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _) wordCounts.print() |