aboutsummaryrefslogtreecommitdiff
path: root/docs/streaming-programming-guide.md
Commit message (Collapse)AuthorAgeFilesLines
* [SPARK-1439, SPARK-1440] Generate unified Scaladoc across projects and JavadocsMatei Zaharia2014-04-211-28/+28
| | | | | | | | | | | | | | | | | | | | | | I used the sbt-unidoc plugin (https://github.com/sbt/sbt-unidoc) to create a unified Scaladoc of our public packages, and generate Javadocs as well. One limitation is that I haven't found an easy way to exclude packages in the Javadoc; there is a SBT task that identifies Java sources to run javadoc on, but it's been very difficult to modify it from outside to change what is set in the unidoc package. Some SBT-savvy people should help with this. The Javadoc site also lacks package-level descriptions and things like that, so we may want to look into that. We may decide not to post these right now if it's too limited compared to the Scala one. Example of the built doc site: http://people.csail.mit.edu/matei/spark-unified-docs/ Author: Matei Zaharia <matei@databricks.com> This patch had conflicts when merged, resolved by Committer: Patrick Wendell <pwendell@gmail.com> Closes #457 from mateiz/better-docs and squashes the following commits: a63d4a3 [Matei Zaharia] Skip Java/Scala API docs for Python package 5ea1f43 [Matei Zaharia] Fix links to Java classes in Java guide, fix some JS for scrolling to anchors on page load f05abc0 [Matei Zaharia] Don't include java.lang package names 995e992 [Matei Zaharia] Skip internal packages and class names with $ in JavaDoc a14a93c [Matei Zaharia] typo 76ce64d [Matei Zaharia] Add groups to Javadoc index page, and a first package-info.java ed6f994 [Matei Zaharia] Generate JavaDoc as well, add titles, update doc site to use unified docs acb993d [Matei Zaharia] Add Unidoc plugin for the projects we want Unidoced
* maintain arbitrary state data for each keyChen Chao2014-03-091-1/+1
| | | | | | | | | | RT Author: Chen Chao <crazyjvm@gmail.com> Closes #114 from CrazyJvm/patch-1 and squashes the following commits: dcb0df5 [Chen Chao] maintain arbitrary state data for each key
* SPARK-1173. (#2) Fix typo in Java streaming example.Aaron Kimball2014-03-021-1/+1
| | | | | | | | | | Companion commit to pull request #64, fix the typo on the Java side of the docs. Author: Aaron Kimball <aaron@magnify.io> Closes #65 from kimballa/spark-1173-java-doc-update and squashes the following commits: 8ce11d3 [Aaron Kimball] SPARK-1173. (#2) Fix typo in Java streaming example.
* SPARK-1173. Improve scala streaming docs.Aaron Kimball2014-03-021-5/+33
| | | | | | | | | | | | | Clarify imports to add implicit conversions to DStream and fix other small typos in the streaming intro documentation. Tested by inspecting output via a local jekyll server, c&p'ing the scala commands into a spark terminal. Author: Aaron Kimball <aaron@magnify.io> Closes #64 from kimballa/spark-1173-streaming-docs and squashes the following commits: 6fbff0e [Aaron Kimball] SPARK-1173. Improve scala streaming docs.
* [SPARK-1105] fix site scala version error in docsCodingCat2014-02-191-8/+8
| | | | | | | | | | | | | https://spark-project.atlassian.net/browse/SPARK-1105 fix site scala version error Author: CodingCat <zhunansjtu@gmail.com> Closes #618 from CodingCat/doc_version and squashes the following commits: 39bb8aa [CodingCat] more fixes 65bedb0 [CodingCat] fix site scala version error in doc
* Fix typos in Spark Streaming programming guideAndrew Or2014-02-171-14/+13
| | | | | | | | | | | | Author: Andrew Or <andrewor14@gmail.com> Closes #536 from andrewor14/streaming-typos and squashes the following commits: a05faa6 [Andrew Or] Fix broken link and wording bc2e4bc [Andrew Or] Merge github.com:apache/incubator-spark into streaming-typos d5515b4 [Andrew Or] TD's comments 767ef12 [Andrew Or] Fix broken links 8f4c731 [Andrew Or] Fix typos in programming guide
* Merge pull request #579 from CrazyJvm/patch-1.Chen Chao2014-02-101-1/+1
| | | | | | | | | | | | "in the source DStream" rather than "int the source DStream" "flatMap is a one-to-many DStream operation that creates a new DStream by generating multiple new records from each record int the source DStream." Author: Chen Chao <crazyjvm@gmail.com> Closes #579 and squashes the following commits: 4abcae3 [Chen Chao] in the source DStream
* Merge pull request #497 from tdas/docs-updateTathagata Das2014-01-281-290/+988
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Updated Spark Streaming Programming Guide Here is the updated version of the Spark Streaming Programming Guide. This is still a work in progress, but the major changes are in place. So feedback is most welcome. In general, I have tried to make the guide to easier to understand even if the reader does not know much about Spark. The updated website is hosted here - http://www.eecs.berkeley.edu/~tdas/spark_docs/streaming-programming-guide.html The major changes are: - Overview illustrates the usecases of Spark Streaming - various input sources and various output sources - An example right after overview to quickly give an idea of what Spark Streaming program looks like - Made Java API and examples a first class citizen like Scala by using tabs to show both Scala and Java examples (similar to AMPCamp tutorial's code tabs) - Highlighted the DStream operations updateStateByKey and transform because of their powerful nature - Updated driver node failure recovery text to highlight automatic recovery in Spark standalone mode - Added information about linking and using the external input sources like Kafka and Flume - In general, reorganized the sections to better show the Basic section and the more advanced sections like Tuning and Recovery. Todos: - Links to the docs of external Kafka, Flume, etc - Illustrate window operation with figure as well as example. Author: Tathagata Das <tathagata.das1565@gmail.com> == Merge branch commits == commit 18ff10556570b39d672beeb0a32075215cfcc944 Author: Tathagata Das <tathagata.das1565@gmail.com> Date: Tue Jan 28 21:49:30 2014 -0800 Fixed a lot of broken links. commit 34a5a6008dac2e107624c7ff0db0824ee5bae45f Author: Tathagata Das <tathagata.das1565@gmail.com> Date: Tue Jan 28 18:02:28 2014 -0800 Updated github url to use SPARK_GITHUB_URL variable. commit f338a60ae8069e0a382d2cb170227e5757cc0b7a Author: Tathagata Das <tathagata.das1565@gmail.com> Date: Mon Jan 27 22:42:42 2014 -0800 More updates based on Patrick and Harvey's comments. commit 89a81ff25726bf6d26163e0dd938290a79582c0f Author: Tathagata Das <tathagata.das1565@gmail.com> Date: Mon Jan 27 13:08:34 2014 -0800 Updated docs based on Patricks PR comments. commit d5b6196b532b5746e019b959a79ea0cc013a8fc3 Author: Tathagata Das <tathagata.das1565@gmail.com> Date: Sun Jan 26 20:15:58 2014 -0800 Added spark.streaming.unpersist config and info on StreamingListener interface. commit e3dcb46ab83d7071f611d9b5008ba6bc16c9f951 Author: Tathagata Das <tathagata.das1565@gmail.com> Date: Sun Jan 26 18:41:12 2014 -0800 Fixed docs on StreamingContext.getOrCreate. commit 6c29524639463f11eec721e4d17a9d7159f2944b Author: Tathagata Das <tathagata.das1565@gmail.com> Date: Thu Jan 23 18:49:39 2014 -0800 Added example and figure for window operations, and links to Kafka and Flume API docs. commit f06b964a51bb3b21cde2ff8bdea7d9785f6ce3a9 Author: Tathagata Das <tathagata.das1565@gmail.com> Date: Wed Jan 22 22:49:12 2014 -0800 Fixed missing endhighlight tag in the MLlib guide. commit 036a7d46187ea3f2a0fb8349ef78f10d6c0b43a9 Merge: eab351d a1cd185 Author: Tathagata Das <tathagata.das1565@gmail.com> Date: Wed Jan 22 22:17:42 2014 -0800 Merge remote-tracking branch 'apache/master' into docs-update commit eab351d05c0baef1d4b549e1581310087158d78d Author: Tathagata Das <tathagata.das1565@gmail.com> Date: Wed Jan 22 22:17:15 2014 -0800 Update Spark Streaming Programming Guide.
* Fixed loose ends in docs.Tathagata Das2014-01-141-2/+2
|
* Removed StreamingContext.registerInputStream and registerOutputStream - they ↵Tathagata Das2014-01-131-15/+17
| | | | were useless as InputDStream has been made to register itself. Also made DStream.register() private[streaming] - not useful to expose the confusing function. Updated a lot of documentation.
* Merge remote-tracking branch 'apache/master' into dstream-moveTathagata Das2014-01-121-2/+2
|\ | | | | | | | | Conflicts: streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala
| * Rename DStream.foreach to DStream.foreachRDDPatrick Wendell2014-01-121-2/+2
| | | | | | | | | | | | `foreachRDD` makes it clear that the granularity of this operator is per-RDD. As it stands, `foreach` is inconsistent with with `map`, `filter`, and the other DStream operators which get pushed down to individual records within each RDD.
* | Moved DStream, DStreamCheckpointData and PairDStream from ↵Tathagata Das2014-01-121-1/+1
|/ | | | org.apache.spark.streaming to org.apache.spark.streaming.dstream.
* run-example -> bin/run-examplePrashant Sharma2014-01-021-2/+2
|
* Updated docs for SparkConf and handled review commentsMatei Zaharia2013-12-301-2/+2
|
* Various broken links in documentationPatrick Wendell2013-12-071-4/+4
|
* Add a `repartition` operator.Patrick Wendell2013-10-241-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds an operator called repartition with more straightforward semantics than the current `coalesce` operator. There are a few use cases where this operator is useful: 1. If a user wants to increase the number of partitions in the RDD. This is more common now with streaming. E.g. a user is ingesting data on one node but they want to add more partitions to ensure parallelism of subsequent operations across threads or the cluster. Right now they have to call rdd.coalesce(numSplits, shuffle=true) - that's super confusing. 2. If a user has input data where the number of partitions is not known. E.g. > sc.textFile("some file").coalesce(50).... This is both vague semantically (am I growing or shrinking this RDD) but also, may not work correctly if the base RDD has fewer than 50 partitions. The new operator forces shuffles every time, so it will always produce exactly the number of new partitions. It also throws an exception rather than silently not-working if a bad input is passed. I am currently adding streaming tests (requires refactoring some of the test suite to allow testing at partition granularity), so this is not ready for merge yet. But feedback is welcome.
* Add docs for standalone scheduler fault toleranceAaron Davidson2013-10-081-3/+2
| | | | Also fix a couple HTML/Markdown issues in other files.
* More fixesMatei Zaharia2013-09-011-2/+10
|
* Fix more URLs in docsMatei Zaharia2013-09-011-3/+3
|
* Update docs for new packageMatei Zaharia2013-09-011-14/+14
|
* Change build and run instructions to use assembliesMatei Zaharia2013-08-291-2/+2
| | | | | | | | | | | | | | | | This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
* Linking custom receiver guidePrashant Sharma2013-08-231-0/+3
|
* Fixes typos in Spark Streaming Programming GuideAndy Konwinski2013-07-121-2/+2
| | | These typos were reported on the spark-users mailing list, see: https://groups.google.com/d/msg/spark-users/SyLGgJlKCrI/LpeBypOkSMUJ
* Typos: cluser -> clusterAndrew Ash2013-04-101-2/+2
|
* More doc tweaksMatei Zaharia2013-02-261-0/+1
|
* Merge pull request #500 from pwendell/streaming-docsTathagata Das2013-02-251-2/+2
|\ | | | | Minor changes based on feedback
| * meta-dataPatrick Wendell2013-02-251-1/+1
| |
| * One more change done with TDPatrick Wendell2013-02-251-1/+1
| |
| * Minor changes based on feedbackPatrick Wendell2013-02-251-2/+2
| |
* | Merge branch 'master' of github.com:mesos/sparkMatei Zaharia2013-02-251-4/+6
|\|
| * Some changes to streaming failure docs.Patrick Wendell2013-02-251-4/+6
| | | | | | | | | | | | TD gave me the go-ahead to just make these changes: - Define stateful dstream - Some minor wording fixes
* | Allow passing sparkHome and JARs to StreamingContext constructorMatei Zaharia2013-02-251-7/+3
| | | | | | | | | | Also warns if spark.cleaner.ttl is not set in the version where you pass your own SparkContext.
* | Some tweaks to docsMatei Zaharia2013-02-251-5/+5
|/
* Fixed class paths and dependencies based on Matei's comments.Tathagata Das2013-02-241-3/+3
|
* Updated streaming programming guide with Java API info, and comments from ↵Tathagata Das2013-02-231-11/+74
| | | | Patrick.
* Change spark.cleaner.delay to spark.cleaner.ttl. Updated docs.Tathagata Das2013-02-231-1/+1
|
* Changed networkStream to socketStream and pluggableNetworkStream to become ↵Tathagata Das2013-02-181-5/+5
| | | | networkStream as a way to create streams from arbitrary network receiver.
* Added checkpointing and fault-tolerance semantics to the programming guide. ↵Tathagata Das2013-02-181-52/+194
| | | | Fixed default checkpoint interval to being a multiple of slide duration. Fixed visibility of some classes and objects to clean up docs.
* Added documentation for PairDStreamFunctions.Tathagata Das2013-01-131-20/+25
|
* Renamed examples and added documentation.Tathagata Das2013-01-071-7/+7
|
* Updated Streaming Programming Guide.Tathagata Das2013-01-011-13/+154
|
* Improved jekyll and scala docs. Made many classes and method private to ↵Tathagata Das2012-12-291-26/+30
| | | | remove them from scala docs.
* Streaming programming guide. STREAMING-2 #resolvePatrick Wendell2012-11-131-0/+163
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-10 00:40:43 +0000