aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
| | * | | | | | ZeroMQ stream as receiverPrashant Sharma2013-02-194-0/+115
| | | |/ / / / | | |/| | | |
| * | | | | | Merge pull request #487 from tdas/streamingTathagata Das2013-02-20113-903/+1387
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | Merged latest master changes to streaming
| | | | | * | | Fixed condition in InputDStream isTimeValid.Tathagata Das2013-02-211-1/+1
| | | |_|/ / / | | |/| | | |
| | * | | | | Merge branch 'mesos-streaming' into streamingTathagata Das2013-02-207-50/+103
| | |\ \ \ \ \ | | |/ / / / / | |/| | | | | | | | | | | | | | | | | | | Conflicts: streaming/src/test/java/spark/streaming/JavaAPISuite.java
| * | | | | | Merge pull request #486 from ScrapCodes/akka-example-bug-fixTathagata Das2013-02-201-1/+1
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | A bug fix to AkkaWordCount example.
| | * | | | | | A bug fix post merge, following changes to AkkaUtilsPrashant Sharma2013-02-201-1/+1
| | | |/ / / / | | |/| | | |
| * | | | | | Merge pull request #481 from pwendell/stream-rdd-type-streamingTathagata Das2013-02-196-19/+146
| |\ \ \ \ \ \ | | |/ / / / / | |/| | | | | STREAMING-51: Add RDD type as a type parameter in JavaDStreamLike Edit (streaming/ version)
| | * | | | | Small changes that were missing in mergePatrick Wendell2013-02-192-0/+2
| | | | | | |
| | * | | | | Use RDD type for `slice` operator in Java.Patrick Wendell2013-02-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit uses the RDD type in `slice`, making it available to both normal and pair RDD's in java. It also updates the signature for `slice` to match changes in the Scala API.
| | * | | | | Use RDD type for `transform` operator in Java.Patrick Wendell2013-02-192-7/+122
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is an improved implementation of the `transform` operator in Java. The main difference is that this allows all four possible types of transform functions 1. JavaRDD -> JavaRDD 2. JavaRDD -> JavaPairRDD 3. JavaPairRDD -> JavaPairRDD 4. JavaPairRDD -> JavaRDD whereas previously only (1) and (3) were possible. Conflicts: streaming/src/test/java/spark/streaming/JavaAPISuite.java
| | * | | | | Use RDD type for `foreach` operator in Java.Patrick Wendell2013-02-196-11/+21
| |/ / / / /
| * | | | | Merge pull request #476 from tdas/streamingTathagata Das2013-02-1952-862/+2142
| |\ \ \ \ \ | | | | | | | | | | | | | | Major modifications to fix driver fault-tolerance with file input stream
| | | * | | | Fixed bug in CheckpointSuiteTathagata Das2013-02-202-6/+6
| | | | | | |
| | | * | | | Merge branch 'mesos-master' into streamingTathagata Das2013-02-206-8/+51
| | | |\ \ \ \
| | | * | | | | Merge branch 'mesos-master' into streamingTathagata Das2013-02-20109-893/+1408
| | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/rdd/CheckpointRDD.scala streaming/src/main/scala/spark/streaming/dstream/ReducedWindowedDStream.scala
| | * | | | | | Added comment to the KafkaWordCount, given by Sean McNamara.Tathagata Das2013-02-191-0/+12
| | | | | | | |
| | * | | | | | Merge branch 'mesos-streaming' into streamingTathagata Das2013-02-190-0/+0
| | |\ \ \ \ \ \ | | |/ / / / / / | |/| | | | | |
| * | | | | | | Merge pull request #455 from tdas/streamingTathagata Das2013-02-07183-2473/+4256
| |\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | Merging latest master branch changes to the streaming branch
| | | * \ \ \ \ \ Merge branch 'streaming' into ScrapCodes-streaming-actorTathagata Das2013-02-19209-3356/+5876
| | | |\ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: docs/plugin-custom-receiver.md streaming/src/main/scala/spark/streaming/StreamingContext.scala streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala streaming/src/main/scala/spark/streaming/dstream/PluggableInputDStream.scala streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala streaming/src/test/scala/spark/streaming/InputStreamsSuite.scala
| | | | * | | | | | Changed networkStream to socketStream and pluggableNetworkStream to become ↵Tathagata Das2013-02-189-35/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | networkStream as a way to create streams from arbitrary network receiver.
| | | | * | | | | | Merge branch 'streaming' into ScrapCode-streamingTathagata Das2013-02-18242-3438/+9569
| | | | |\ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala streaming/src/main/scala/spark/streaming/dstream/NetworkInputDStream.scala
| | | | | * | | | | | Added checkpointing and fault-tolerance semantics to the programming guide. ↵Tathagata Das2013-02-188-59/+206
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed default checkpoint interval to being a multiple of slide duration. Fixed visibility of some classes and objects to clean up docs.
| | | | | * | | | | | Many changes to ensure better 2nd recovery if 2nd failure happens whileTathagata Das2013-02-1718-97/+208
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | recovering from 1st failure - Made the scheduler to checkpoint after clearing old metadata which ensures that a new checkpoint is written as soon as at least one batch gets computed while recovering from a failure. This ensures that if there is a 2nd failure while recovering from 1st failure, the system start 2nd recovery from a newer checkpoint. - Modified Checkpoint writer to write checkpoint in a different thread. - Added a check to make sure that compute for InputDStreams gets called only for strictly increasing times. - Changed implementation of slice to call getOrCompute on parent DStream in time-increasing order. - Added testcase to test slice. - Fixed testGroupByKeyAndWindow testcase in JavaAPISuite to verify results with expected output in an order-independent manner.
| | | | | * | | | | | Made MasterFailureTest more robust.Tathagata Das2013-02-151-4/+22
| | | | | | | | | | |
| | | | | * | | | | | Moved Java streaming examples to examples/src/main/java/spark/streaming/... ↵Tathagata Das2013-02-144-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | and fixed logging in NetworkInputTracker to highlight errors when receiver deregisters/shuts down.
| | | | | * | | | | | Added TwitterInputDStream from example to StreamingContext. Renamed example ↵Tathagata Das2013-02-144-45/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | TwitterBasic to TwitterPopularTags.
| | | | | * | | | | | Removed countByKeyAndWindow on paired DStreams, and added ↵Tathagata Das2013-02-149-189/+231
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | countByValueAndWindow for all DStreams. Updated both scala and java API and testsuites.
| | | | | * | | | | | Changes functions comments to make them more consistent.Tathagata Das2013-02-132-45/+45
| | | | | | | | | | |
| | | | | * | | | | | Added filter functionality to reduceByKeyAndWindow with inverse. ↵Tathagata Das2013-02-137-81/+102
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Consolidated reduceByKeyAndWindow's many functions into smaller number of functions with optional parameters.
| | | | | * | | | | | Changed scheduler and file input stream to fix bugs in the driver fault ↵Tathagata Das2013-02-1318-452/+693
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | tolerance. Added MasterFailureTest to rigorously test master fault tolerance with file input stream.
| | | | | * | | | | | Fixed bugs in FileInputDStream and Scheduler that occasionally failed to ↵Tathagata Das2013-02-106-91/+221
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | reprocess old files after recovering from master failure. Completely modified spark.streaming.FailureTest to test multiple master failures using file input stream.
| | | | | * | | | | | Fixed bug in CheckpointRDD to prevent exception when the original RDD had ↵Tathagata Das2013-02-102-2/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | zero splits.
| | | | | * | | | | | Added an initial spark job to ensure worker nodes are initialized.Tathagata Das2013-02-092-2/+7
| | | |_|/ / / / / / | | |/| | | | | | |
| | * | | | | | | | Merge branch 'mesos-master' into streamingTathagata Das2013-02-07177-2220/+3876
| | |\ \ \ \ \ \ \ \
| | * | | | | | | | | Updated JavaStreamingContext with updated kafkaStream API.Tathagata Das2013-02-071-17/+9
| | | | | | | | | | |
| | * | | | | | | | | Merge branch 'mesos-streaming' into streamingTathagata Das2013-02-077-96/+102
| | |\ \ \ \ \ \ \ \ \ | | |/ / / / / / / / / | |/| | | | | | | | |
| * | | | | | | | | | Merge pull request #372 from Reinvigorate/sm-kafkaTathagata Das2013-02-074-99/+24
| |\ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Removing offset management code that is non-existent in kafka 0.7.0+
| | * | | | | | | | | | kafkaStream API cleanup. A quorum of zookeepers can now be specifiedseanm2013-01-183-23/+18
| | | | | | | | | | | |
| | * | | | | | | | | | further KafkaInputDStream cleanup (removing unused and commented out code ↵seanm2013-01-181-69/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | relating to offset management)
| | * | | | | | | | | | kafka jar wasn't being included by run scriptseanm2013-01-181-0/+3
| | | | | | | | | | | |
| | * | | | | | | | | | Removing offset management code that is non-existent in kafka 0.7.0+seanm2013-01-141-7/+0
| | | | | | | | | | | |
| * | | | | | | | | | | Merge pull request #373 from Reinvigorate/sm-updateStateByKeyTathagata Das2013-02-074-6/+78
| |\ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | StateDStream changes to give updateStateByKey consistent behavior
| | * | | | | | | | | | | Splitting StreamingContext.queueStream into two methodsseanm2013-01-201-4/+18
| | | | | | | | | | | | |
| | * | | | | | | | | | | adding updateStateByKey object lifecycle testseanm2013-01-202-0/+50
| | | | | | | | | | | | |
| | * | | | | | | | | | | Merge branch 'streaming' into sm-updateStateByKeyseanm2013-01-154-11/+11
| | |\ \ \ \ \ \ \ \ \ \ \
| | * | | | | | | | | | | | StateDStream changes to give updateStateByKey consistent behaviorseanm2013-01-141-2/+10
| | | |/ / / / / / / / / / | | |/| | | | | | | | | |
| | | | * | | | | | | | | Fixed checkpoint testcasesTathagata Das2013-01-233-172/+129
| | | | | | | | | | | | |
| | | | * | | | | | | | | Added support for rescheduling unprocessed batches on master failure.Tathagata Das2013-01-235-12/+53
| | | | | | | | | | | | |
| | | | * | | | | | | | | Added support for saving input files of FileInputDStream to graph ↵Tathagata Das2013-01-226-66/+159
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | checkpoints. Modified 'file input stream with checkpoint' testcase to test recovery of pre-master-failure input files.
| | | | * | | | | | | | | Refactored DStreamCheckpointData.Tathagata Das2013-01-224-64/+99
| | |_|/ / / / / / / / / | |/| | | | | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-24 00:33:33 +0000