| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
| | | |/ / / /
| | |/| | | | |
|
| |\ \ \ \ \ \
| | | | | | | |
| | | | | | | | |
Merged latest master changes to streaming
|
| | | |_|/ / /
| | |/| | | | |
|
| | |\ \ \ \ \
| | |/ / / / /
| |/| | | | |
| | | | | | |
| | | | | | | |
Conflicts:
streaming/src/test/java/spark/streaming/JavaAPISuite.java
|
| |\ \ \ \ \ \
| | | | | | | |
| | | | | | | | |
A bug fix to AkkaWordCount example.
|
| | | |/ / / /
| | |/| | | | |
|
| |\ \ \ \ \ \
| | |/ / / / /
| |/| | | | | |
STREAMING-51: Add RDD type as a type parameter in JavaDStreamLike Edit (streaming/ version)
|
| | | | | | | |
|
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
This commit uses the RDD type in `slice`, making it available to both normal
and pair RDD's in java. It also updates the signature for `slice` to match
changes in the Scala API.
|
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
This is an improved implementation of the `transform` operator in Java.
The main difference is that this allows all four possible types of
transform functions
1. JavaRDD -> JavaRDD
2. JavaRDD -> JavaPairRDD
3. JavaPairRDD -> JavaPairRDD
4. JavaPairRDD -> JavaRDD
whereas previously only (1) and (3) were possible.
Conflicts:
streaming/src/test/java/spark/streaming/JavaAPISuite.java
|
| |/ / / / / |
|
| |\ \ \ \ \
| | | | | | |
| | | | | | | |
Major modifications to fix driver fault-tolerance with file input stream
|
| | | | | | | |
|
| | | |\ \ \ \ |
|
| | |/| | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
Conflicts:
core/src/main/scala/spark/rdd/CheckpointRDD.scala
streaming/src/main/scala/spark/streaming/dstream/ReducedWindowedDStream.scala
|
| | | | | | | | |
|
| | |\ \ \ \ \ \
| | |/ / / / / /
| |/| | | | | | |
|
| |\ \ \ \ \ \ \
| | | | | | | | |
| | | | | | | | | |
Merging latest master branch changes to the streaming branch
|
| | | |\ \ \ \ \ \
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | | |
Conflicts:
docs/plugin-custom-receiver.md
streaming/src/main/scala/spark/streaming/StreamingContext.scala
streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala
streaming/src/main/scala/spark/streaming/dstream/PluggableInputDStream.scala
streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala
streaming/src/test/scala/spark/streaming/InputStreamsSuite.scala
|
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | | |
networkStream as a way to create streams from arbitrary network receiver.
|
| | | | |\ \ \ \ \ \
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | | |
Conflicts:
streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala
streaming/src/main/scala/spark/streaming/dstream/NetworkInputDStream.scala
|
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | | |
Fixed default checkpoint interval to being a multiple of slide duration. Fixed visibility of some classes and objects to clean up docs.
|
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | | |
recovering from 1st failure
- Made the scheduler to checkpoint after clearing old metadata which
ensures that a new checkpoint is written as soon as at least one batch
gets computed while recovering from a failure. This ensures that if
there is a 2nd failure while recovering from 1st failure, the system
start 2nd recovery from a newer checkpoint.
- Modified Checkpoint writer to write checkpoint in a different thread.
- Added a check to make sure that compute for InputDStreams gets called
only for strictly increasing times.
- Changed implementation of slice to call getOrCompute on parent DStream
in time-increasing order.
- Added testcase to test slice.
- Fixed testGroupByKeyAndWindow testcase in JavaAPISuite to verify
results with expected output in an order-independent manner.
|
| | | | | | | | | | | |
|
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | | |
and fixed logging in NetworkInputTracker to highlight errors when receiver deregisters/shuts down.
|
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | | |
TwitterBasic to TwitterPopularTags.
|
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | | |
countByValueAndWindow for all DStreams. Updated both scala and java API and testsuites.
|
| | | | | | | | | | | |
|
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | | |
Consolidated reduceByKeyAndWindow's many functions into smaller number of functions with optional parameters.
|
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | | |
tolerance. Added MasterFailureTest to rigorously test master fault tolerance with file input stream.
|
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | | |
reprocess old files after recovering from master failure. Completely modified spark.streaming.FailureTest to test multiple master failures using file input stream.
|
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | | |
zero splits.
|
| | | |_|/ / / / / /
| | |/| | | | | | | |
|
| | |\ \ \ \ \ \ \ \ |
|
| | | | | | | | | | | |
|
| | |\ \ \ \ \ \ \ \ \
| | |/ / / / / / / / /
| |/| | | | | | | | | |
|
| |\ \ \ \ \ \ \ \ \ \
| | | | | | | | | | | |
| | | | | | | | | | | | |
Removing offset management code that is non-existent in kafka 0.7.0+
|
| | | | | | | | | | | | |
|
| | | | | | | | | | | |
| | | | | | | | | | | |
| | | | | | | | | | | |
| | | | | | | | | | | | |
relating to offset management)
|
| | | | | | | | | | | | |
|
| | | | | | | | | | | | |
|
| |\ \ \ \ \ \ \ \ \ \ \
| | | | | | | | | | | | |
| | | | | | | | | | | | | |
StateDStream changes to give updateStateByKey consistent behavior
|
| | | | | | | | | | | | | |
|
| | | | | | | | | | | | | |
|
| | |\ \ \ \ \ \ \ \ \ \ \ |
|
| | | |/ / / / / / / / / /
| | |/| | | | | | | | | | |
|
| | | | | | | | | | | | | |
|
| | | | | | | | | | | | | |
|
| | | | | | | | | | | | |
| | | | | | | | | | | | |
| | | | | | | | | | | | |
| | | | | | | | | | | | | |
checkpoints. Modified 'file input stream with checkpoint' testcase to test recovery of pre-master-failure input files.
|
| | |_|/ / / / / / / / /
| |/| | | | | | | | | | |
|