aboutsummaryrefslogtreecommitdiff
path: root/streaming
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'streaming' into ScrapCodes-streaming-actorTathagata Das2013-02-1946-1056/+2026
|\ | | | | | | | | | | | | | | | | | | Conflicts: docs/plugin-custom-receiver.md streaming/src/main/scala/spark/streaming/StreamingContext.scala streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala streaming/src/main/scala/spark/streaming/dstream/PluggableInputDStream.scala streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala streaming/src/test/scala/spark/streaming/InputStreamsSuite.scala
| * Changed networkStream to socketStream and pluggableNetworkStream to become ↵Tathagata Das2013-02-184-21/+22
| | | | | | | | networkStream as a way to create streams from arbitrary network receiver.
| * Merge branch 'streaming' into ScrapCode-streamingTathagata Das2013-02-1845-1027/+2015
| |\ | | | | | | | | | | | | | | | Conflicts: streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala streaming/src/main/scala/spark/streaming/dstream/NetworkInputDStream.scala
| | * Added checkpointing and fault-tolerance semantics to the programming guide. ↵Tathagata Das2013-02-186-6/+11
| | | | | | | | | | | | Fixed default checkpoint interval to being a multiple of slide duration. Fixed visibility of some classes and objects to clean up docs.
| | * Many changes to ensure better 2nd recovery if 2nd failure happens whileTathagata Das2013-02-1718-97/+208
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | recovering from 1st failure - Made the scheduler to checkpoint after clearing old metadata which ensures that a new checkpoint is written as soon as at least one batch gets computed while recovering from a failure. This ensures that if there is a 2nd failure while recovering from 1st failure, the system start 2nd recovery from a newer checkpoint. - Modified Checkpoint writer to write checkpoint in a different thread. - Added a check to make sure that compute for InputDStreams gets called only for strictly increasing times. - Changed implementation of slice to call getOrCompute on parent DStream in time-increasing order. - Added testcase to test slice. - Fixed testGroupByKeyAndWindow testcase in JavaAPISuite to verify results with expected output in an order-independent manner.
| | * Made MasterFailureTest more robust.Tathagata Das2013-02-151-4/+22
| | |
| | * Moved Java streaming examples to examples/src/main/java/spark/streaming/... ↵Tathagata Das2013-02-141-1/+1
| | | | | | | | | | | | and fixed logging in NetworkInputTracker to highlight errors when receiver deregisters/shuts down.
| | * Added TwitterInputDStream from example to StreamingContext. Renamed example ↵Tathagata Das2013-02-142-17/+105
| | | | | | | | | | | | TwitterBasic to TwitterPopularTags.
| | * Removed countByKeyAndWindow on paired DStreams, and added ↵Tathagata Das2013-02-148-183/+226
| | | | | | | | | | | | countByValueAndWindow for all DStreams. Updated both scala and java API and testsuites.
| | * Changes functions comments to make them more consistent.Tathagata Das2013-02-132-45/+45
| | |
| | * Added filter functionality to reduceByKeyAndWindow with inverse. ↵Tathagata Das2013-02-137-81/+102
| | | | | | | | | | | | Consolidated reduceByKeyAndWindow's many functions into smaller number of functions with optional parameters.
| | * Changed scheduler and file input stream to fix bugs in the driver fault ↵Tathagata Das2013-02-1318-452/+693
| | | | | | | | | | | | tolerance. Added MasterFailureTest to rigorously test master fault tolerance with file input stream.
| | * Fixed bugs in FileInputDStream and Scheduler that occasionally failed to ↵Tathagata Das2013-02-106-91/+221
| | | | | | | | | | | | reprocess old files after recovering from master failure. Completely modified spark.streaming.FailureTest to test multiple master failures using file input stream.
| | * Added an initial spark job to ensure worker nodes are initialized.Tathagata Das2013-02-092-2/+7
| | |
| | * Merge branch 'mesos-master' into streamingTathagata Das2013-02-0722-15/+195
| | |\
| | | * Streaming constructor which takes JavaSparkContextPatrick Wendell2013-02-051-0/+8
| | | | | | | | | | | | | | | | | | | | It's sometimes helpful to directly pass a JavaSparkContext, and take advantage of the various constructors available for that.
| | | * Remove activation of profiles by defaultMikhail Bautin2013-01-311-11/+0
| | | | | | | | | | | | | | | | | | | | See the discussion at https://github.com/mesos/spark/pull/355 for why default profile activation is a problem.
| | | * Merge pull request #415 from stephenh/driverMatei Zaharia2013-01-297-8/+8
| | | |\ | | | | | | | | | | Replace old 'master' term with 'driver'.
| | | | * Replace old 'master' term with 'driver'.Stephen Haberman2013-01-257-8/+8
| | | | |
| | | * | Fix code that depended on metadata cleaner interval being in minutesMatei Zaharia2013-01-282-5/+5
| | | |/
| | | * Refactor daemon thread pool creation.Josh Rosen2013-01-211-2/+3
| | | |
| | | * Move JavaAPISuite into spark.streaming.Stephen Haberman2013-01-212-0/+0
| | | |
| | | * Add Maven build file for streaming, and fix some issues in SBT fileMatei Zaharia2013-01-2010-0/+182
| | | | | | | | | | | | | | | | | | | | | | | | As part of this, changed our Scala 2.9.2 Kafka library to be available as a local Maven repository, following the example in (http://blog.dub.podval.org/2010/01/maven-in-project-repository.html)
| | * | Updated JavaStreamingContext with updated kafkaStream API.Tathagata Das2013-02-071-17/+9
| | | |
| | * | Merge branch 'mesos-streaming' into streamingTathagata Das2013-02-075-88/+91
| | |\ \
| | | * \ Merge pull request #372 from Reinvigorate/sm-kafkaTathagata Das2013-02-072-91/+13
| | | |\ \ | | | | | | | | | | | | Removing offset management code that is non-existent in kafka 0.7.0+
| | | | * | kafkaStream API cleanup. A quorum of zookeepers can now be specifiedseanm2013-01-182-15/+10
| | | | | |
| | | | * | further KafkaInputDStream cleanup (removing unused and commented out code ↵seanm2013-01-181-69/+3
| | | | | | | | | | | | | | | | | | | | | | | | relating to offset management)
| | | | * | Removing offset management code that is non-existent in kafka 0.7.0+seanm2013-01-141-7/+0
| | | | | |
| | | * | | Merge pull request #373 from Reinvigorate/sm-updateStateByKeyTathagata Das2013-02-074-6/+78
| | | |\ \ \ | | | | |_|/ | | | |/| | StateDStream changes to give updateStateByKey consistent behavior
| | | | * | Splitting StreamingContext.queueStream into two methodsseanm2013-01-201-4/+18
| | | | | |
| | | | * | adding updateStateByKey object lifecycle testseanm2013-01-202-0/+50
| | | | | |
| | | | * | Merge branch 'streaming' into sm-updateStateByKeyseanm2013-01-153-8/+8
| | | | |\ \
| | | | * | | StateDStream changes to give updateStateByKey consistent behaviorseanm2013-01-141-2/+10
| | | | | |/ | | | | |/|
| | * | | | Fixed checkpoint testcasesTathagata Das2013-01-233-172/+129
| | | | | |
| | * | | | Added support for rescheduling unprocessed batches on master failure.Tathagata Das2013-01-235-12/+53
| | | | | |
| | * | | | Added support for saving input files of FileInputDStream to graph ↵Tathagata Das2013-01-226-66/+159
| | | | | | | | | | | | | | | | | | | | | | | | checkpoints. Modified 'file input stream with checkpoint' testcase to test recovery of pre-master-failure input files.
| | * | | | Refactored DStreamCheckpointData.Tathagata Das2013-01-224-64/+99
| | |/ / /
| * | | | Plug in actor as stream receiver APIPrashant Sharma2013-01-193-0/+152
| | | | |
| * | | | Changed method name of createReceiver to getReceiver as it is not intended ↵Prashant Sharma2013-01-196-28/+28
| | | | | | | | | | | | | | | | | | | | to be a factory.
* | | | | actor as receiverPrashant Sharma2013-01-224-0/+269
| | | | |
* | | | | Changed method name of createReceiver to getReceiver as it is not intended ↵Prashant Sharma2013-01-216-8/+8
| |/ / / |/| | | | | | | | | | | to be a factory.
* | | | Fixed streaming testsuite bugsTathagata Das2013-01-207-6/+24
| | | |
* | | | Merge branch 'mesos-streaming' into streamingTathagata Das2013-01-208-120/+2452
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/api/java/JavaRDDLike.scala core/src/main/scala/spark/api/java/JavaSparkContext.scala core/src/test/scala/spark/JavaAPISuite.java
| * | | Moving tests to appropriate directoryPatrick Wendell2013-01-172-0/+0
| | | |
| * | | Adding queueStream and some slight refactoringPatrick Wendell2013-01-172-81/+163
| | | |
| * | | Merge branch 'streaming' into streaming-java-apiPatrick Wendell2013-01-171-3/+3
| |\ \ \ | | | |/ | | |/|
| * | | Import fixupPatrick Wendell2013-01-173-3/+0
| | | |
| * | | Style cleanupPatrick Wendell2013-01-173-3/+26
| | | |
| * | | Checkpointing in Streaming java APIPatrick Wendell2013-01-173-3/+93
| | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-20 06:20:56 +0000