aboutsummaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorTathagata Das <tathagata.das1565@gmail.com>2013-02-24 16:32:44 -0800
committerTathagata Das <tathagata.das1565@gmail.com>2013-02-24 16:32:44 -0800
commitabb54718658f8cfa0005fe99f4159eabce074f45 (patch)
treeec822b1e0fdab2eccabad2437e20dca8371230cf /docs
parent5ab37be9831e8a70b2502b14aed1c87cb002a189 (diff)
downloadspark-abb54718658f8cfa0005fe99f4159eabce074f45.tar.gz
spark-abb54718658f8cfa0005fe99f4159eabce074f45.tar.bz2
spark-abb54718658f8cfa0005fe99f4159eabce074f45.zip
Removing duplicate doc.
Diffstat (limited to 'docs')
-rw-r--r--docs/custom-streaming-receiver.md101
-rw-r--r--docs/streaming-custom-receivers.md4
2 files changed, 2 insertions, 103 deletions
diff --git a/docs/custom-streaming-receiver.md b/docs/custom-streaming-receiver.md
deleted file mode 100644
index 0eb4246158..0000000000
--- a/docs/custom-streaming-receiver.md
+++ /dev/null
@@ -1,101 +0,0 @@
----
-layout: global
-title: Tutorial - Spark streaming, Plugging in a custom receiver.
----
-
-A "Spark streaming" receiver can be a simple network stream, streams of messages from a message queue, files etc. A receiver can also assume roles more than just receiving data like filtering, preprocessing, to name a few of the possibilities. The api to plug-in any user defined custom receiver is thus provided to encourage development of receivers which may be well suited to ones specific need.
-
-This guide shows the programming model and features by walking through a simple sample receiver and corresponding Spark Streaming application.
-
-
-## A quick and naive walk-through
-
-### Write a simple receiver
-
-This starts with implementing [Actor](#References)
-
-Following is a simple socket text-stream receiver, which is appearently overly simplified using Akka's socket.io api.
-
-{% highlight scala %}
-
- class SocketTextStreamReceiver (host:String,
- port:Int,
- bytesToString: ByteString => String) extends Actor with Receiver {
-
- override def preStart = IOManager(context.system).connect(host, port)
-
- def receive = {
- case IO.Read(socket, bytes) => pushBlock(bytesToString(bytes))
- }
-
- }
-
-
-{% endhighlight %}
-
-All we did here is mixed in trait Receiver and called pushBlock api method to push our blocks of data. Please refer to scala-docs of Receiver for more details.
-
-### A sample spark application
-
-* First create a Spark streaming context with master url and batchduration.
-
-{% highlight scala %}
-
- val ssc = new StreamingContext(master, "WordCountCustomStreamSource",
- Seconds(batchDuration))
-
-{% endhighlight %}
-
-* Plug-in the actor configuration into the spark streaming context and create a DStream.
-
-{% highlight scala %}
-
- val lines = ssc.actorStream[String](Props(new SocketTextStreamReceiver(
- "localhost",8445, z => z.utf8String)),"SocketReceiver")
-
-{% endhighlight %}
-
-* Process it.
-
-{% highlight scala %}
-
- val words = lines.flatMap(_.split(" "))
- val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
-
- wordCounts.print()
- ssc.start()
-
-
-{% endhighlight %}
-
-* After processing it, stream can be tested using the netcat utility.
-
- $ nc -l localhost 8445
- hello world
- hello hello
-
-
-## Multiple homogeneous/heterogeneous receivers.
-
-A DStream union operation is provided for taking union on multiple input streams.
-
-{% highlight scala %}
-
- val lines = ssc.actorStream[String](Props(new SocketTextStreamReceiver(
- "localhost",8445, z => z.utf8String)),"SocketReceiver")
-
- // Another socket stream receiver
- val lines2 = ssc.actorStream[String](Props(new SocketTextStreamReceiver(
- "localhost",8446, z => z.utf8String)),"SocketReceiver")
-
- val union = lines.union(lines2)
-
-{% endhighlight %}
-
-Above stream can be easily process as described earlier.
-
-_A more comprehensive example is provided in the spark streaming examples_
-
-## References
-
-1.[Akka Actor documentation](http://doc.akka.io/docs/akka/2.0.5/scala/actors.html)
diff --git a/docs/streaming-custom-receivers.md b/docs/streaming-custom-receivers.md
index 0eb4246158..5476c00d02 100644
--- a/docs/streaming-custom-receivers.md
+++ b/docs/streaming-custom-receivers.md
@@ -1,9 +1,9 @@
---
layout: global
-title: Tutorial - Spark streaming, Plugging in a custom receiver.
+title: Tutorial - Spark Streaming, Plugging in a custom receiver.
---
-A "Spark streaming" receiver can be a simple network stream, streams of messages from a message queue, files etc. A receiver can also assume roles more than just receiving data like filtering, preprocessing, to name a few of the possibilities. The api to plug-in any user defined custom receiver is thus provided to encourage development of receivers which may be well suited to ones specific need.
+A "Spark Streaming" receiver can be a simple network stream, streams of messages from a message queue, files etc. A receiver can also assume roles more than just receiving data like filtering, preprocessing, to name a few of the possibilities. The api to plug-in any user defined custom receiver is thus provided to encourage development of receivers which may be well suited to ones specific need.
This guide shows the programming model and features by walking through a simple sample receiver and corresponding Spark Streaming application.