| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
| |
|
|
|
|
|
| |
This also refactors out the common "gotNext" iterator pattern into
a shared utility class.
|
|
|
|
| |
choose an available port for TestServer.
|
|\
| |
| | |
createNewSparkContext should use sparkHome/jars/environment.
|
| |
| |
| |
| | |
This fixes a bug introduced by Matei's recent change.
|
|/ |
|
|
|
|
| |
controlled centrally.
|
|
|
|
|
|
| |
For some reason, ScalaDoc complained about no such constructor for
StreamingContext; it doesn't seem like an actual Scala error but it
prevented sbt publish and from working because docs weren't built.
|
|
|
|
|
| |
Also warns if spark.cleaner.ttl is not set in the version where you pass
your own SparkContext.
|
| |
|
|
|
|
| |
Patrick.
|
|
|
|
| |
Change rawNetworkStream to rawSocketStream, and added twitter, actor, zeroMQ streams to JavaStreamingContext. Also added them to JavaAPISuite.
|
|\ |
|
| |\
| | |
| | | |
Zeromq streaming
|
| | | |
|
| | | |
|
| | | |
|
|/ / |
|
|\ \
| | |
| | |
| | |
| | | |
Conflicts:
streaming/src/test/java/spark/streaming/JavaAPISuite.java
|
| | | |
|
| | |
| | |
| | |
| | |
| | |
| | | |
This commit uses the RDD type in `slice`, making it available to both normal
and pair RDD's in java. It also updates the signature for `slice` to match
changes in the Scala API.
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This is an improved implementation of the `transform` operator in Java.
The main difference is that this allows all four possible types of
transform functions
1. JavaRDD -> JavaRDD
2. JavaRDD -> JavaPairRDD
3. JavaPairRDD -> JavaPairRDD
4. JavaPairRDD -> JavaRDD
whereas previously only (1) and (3) were possible.
Conflicts:
streaming/src/test/java/spark/streaming/JavaAPISuite.java
|
| |/ |
|
|\ \
| |/
|/|
| |
| |
| | |
Conflicts:
core/src/main/scala/spark/rdd/CheckpointRDD.scala
streaming/src/main/scala/spark/streaming/dstream/ReducedWindowedDStream.scala
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| | |
This ports a useful workaround (the `transform` function) to
JavaPairDStream. It is necessary to do things like sorting which
are not supported yet in the core streaming API.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
|\ \
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Conflicts:
docs/plugin-custom-receiver.md
streaming/src/main/scala/spark/streaming/StreamingContext.scala
streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala
streaming/src/main/scala/spark/streaming/dstream/PluggableInputDStream.scala
streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala
streaming/src/test/scala/spark/streaming/InputStreamsSuite.scala
|
| | |
| | |
| | |
| | | |
networkStream as a way to create streams from arbitrary network receiver.
|
| |\ \
| | | |
| | | |
| | | |
| | | |
| | | | |
Conflicts:
streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala
streaming/src/main/scala/spark/streaming/dstream/NetworkInputDStream.scala
|
| | | |
| | | |
| | | |
| | | | |
Fixed default checkpoint interval to being a multiple of slide duration. Fixed visibility of some classes and objects to clean up docs.
|
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
recovering from 1st failure
- Made the scheduler to checkpoint after clearing old metadata which
ensures that a new checkpoint is written as soon as at least one batch
gets computed while recovering from a failure. This ensures that if
there is a 2nd failure while recovering from 1st failure, the system
start 2nd recovery from a newer checkpoint.
- Modified Checkpoint writer to write checkpoint in a different thread.
- Added a check to make sure that compute for InputDStreams gets called
only for strictly increasing times.
- Changed implementation of slice to call getOrCompute on parent DStream
in time-increasing order.
- Added testcase to test slice.
- Fixed testGroupByKeyAndWindow testcase in JavaAPISuite to verify
results with expected output in an order-independent manner.
|
| | | | |
|
| | | |
| | | |
| | | |
| | | | |
and fixed logging in NetworkInputTracker to highlight errors when receiver deregisters/shuts down.
|
| | | |
| | | |
| | | |
| | | | |
TwitterBasic to TwitterPopularTags.
|
| | | |
| | | |
| | | |
| | | | |
countByValueAndWindow for all DStreams. Updated both scala and java API and testsuites.
|
| | | | |
|
| | | |
| | | |
| | | |
| | | | |
Consolidated reduceByKeyAndWindow's many functions into smaller number of functions with optional parameters.
|
| | | |
| | | |
| | | |
| | | | |
tolerance. Added MasterFailureTest to rigorously test master fault tolerance with file input stream.
|
| | | |
| | | |
| | | |
| | | | |
reprocess old files after recovering from master failure. Completely modified spark.streaming.FailureTest to test multiple master failures using file input stream.
|
| | | | |
|
| | |\| |
|
| | | |
| | | |
| | | |
| | | |
| | | | |
It's sometimes helpful to directly pass a JavaSparkContext,
and take advantage of the various constructors available for that.
|