aboutsummaryrefslogtreecommitdiff
path: root/streaming/src/test
Commit message (Collapse)AuthorAgeFilesLines
* Move some classes to more appropriate packages:Matei Zaharia2013-09-011-2/+3
| | | | | | * RDD, *RDDFunctions -> org.apache.spark.rdd * Utils, ClosureCleaner, SizeEstimator -> org.apache.spark.util * JavaSerializer, KryoSerializer -> org.apache.spark.serializer
* Initial work to rename package to org.apache.sparkMatei Zaharia2013-09-018-47/+48
|
* Add Apache license headers and LICENSE and NOTICE filesMatei Zaharia2013-07-169-2/+155
|
* Initialize Twitter4J OAuth from system properties instead of promptingMatei Zaharia2013-06-291-1/+1
|
* Merge branch 'master' into streamingTathagata Das2013-06-244-14/+25
|\ | | | | | | | | Conflicts: .gitignore
| * Remove debug statementsMridul Muralidharan2013-04-291-2/+0
| |
| * Attempt to fix streaming test failures after yarn branch mergeMridul Muralidharan2013-04-285-1/+8
| |
| * Move streaming test initialization into 'before' blocksJey Kottalam2013-03-282-4/+8
| |
| * Instead of failing to bind to a fixed, already-in-use port, let the OS ↵Mark Hamstra2013-03-011-8/+10
| | | | | | | | choose an available port for TestServer.
| * Changed Flume test to use the same port as other tests, so that can be ↵Tathagata Das2013-02-251-2/+2
| | | | | | | | controlled centrally.
* | Merge pull request #571 from Reinvigorate/sm-kafka-serializersTathagata Das2013-06-242-4/+19
|\ \ | | | | | | Surfacing decoders on KafkaInputDStream
| * | fixing kafkaStream Java API and adding testseanm2013-05-101-0/+6
| | |
| * | adding kafkaStream API testsseanm2013-05-102-2/+13
| | |
| * | Surfacing decoders on KafkaInputDStreamseanm2013-04-161-4/+2
| |/
* / fixing Spark Streaming count() so that 0 will be emitted when there is ↵seanm2013-04-151-2/+2
|/ | | | nothing to count
* Fixed differences in APIs of StreamingContext and JavaStreamingContext. ↵Tathagata Das2013-02-232-6/+33
| | | | Change rawNetworkStream to rawSocketStream, and added twitter, actor, zeroMQ streams to JavaStreamingContext. Also added them to JavaAPISuite.
* Merge branch 'mesos-streaming' into streamingTathagata Das2013-02-202-4/+48
|\ | | | | | | | | Conflicts: streaming/src/test/java/spark/streaming/JavaAPISuite.java
| * Small changes that were missing in mergePatrick Wendell2013-02-191-0/+1
| |
| * Use RDD type for `transform` operator in Java.Patrick Wendell2013-02-191-2/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is an improved implementation of the `transform` operator in Java. The main difference is that this allows all four possible types of transform functions 1. JavaRDD -> JavaRDD 2. JavaRDD -> JavaPairRDD 3. JavaPairRDD -> JavaPairRDD 4. JavaPairRDD -> JavaRDD whereas previously only (1) and (3) were possible. Conflicts: streaming/src/test/java/spark/streaming/JavaAPISuite.java
| * Use RDD type for `foreach` operator in Java.Patrick Wendell2013-02-192-2/+4
| |
* | Merge branch 'mesos-master' into streamingTathagata Das2013-02-201-3/+183
|\ \ | |/ |/| | | | | | | Conflicts: core/src/main/scala/spark/rdd/CheckpointRDD.scala streaming/src/main/scala/spark/streaming/dstream/ReducedWindowedDStream.scala
| * STREAMING-50: Support transform workaround in JavaPairDStreamPatrick Wendell2013-02-121-0/+45
| | | | | | | | | | | | This ports a useful workaround (the `transform` function) to JavaPairDStream. It is necessary to do things like sorting which are not supported yet in the core streaming API.
| * Using tuple swap()Patrick Wendell2013-02-111-2/+2
| |
| * small fixPatrick Wendell2013-02-111-2/+2
| |
| * Fix for MapPartitionsPatrick Wendell2013-02-111-15/+52
| |
| * Fix for flatmapPatrick Wendell2013-02-111-0/+42
| |
| * Indentation fixPatrick Wendell2013-02-111-10/+10
| |
| * Initial cut at replacing K, V in Java filesPatrick Wendell2013-02-111-0/+56
| |
* | Merge branch 'streaming' into ScrapCodes-streaming-actorTathagata Das2013-02-199-440/+443
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: docs/plugin-custom-receiver.md streaming/src/main/scala/spark/streaming/StreamingContext.scala streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala streaming/src/main/scala/spark/streaming/dstream/PluggableInputDStream.scala streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala streaming/src/test/scala/spark/streaming/InputStreamsSuite.scala
| * | Changed networkStream to socketStream and pluggableNetworkStream to become ↵Tathagata Das2013-02-182-4/+3
| | | | | | | | | | | | networkStream as a way to create streams from arbitrary network receiver.
| * | Added checkpointing and fault-tolerance semantics to the programming guide. ↵Tathagata Das2013-02-181-1/+1
| | | | | | | | | | | | Fixed default checkpoint interval to being a multiple of slide duration. Fixed visibility of some classes and objects to clean up docs.
| * | Many changes to ensure better 2nd recovery if 2nd failure happens whileTathagata Das2013-02-177-29/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | recovering from 1st failure - Made the scheduler to checkpoint after clearing old metadata which ensures that a new checkpoint is written as soon as at least one batch gets computed while recovering from a failure. This ensures that if there is a 2nd failure while recovering from 1st failure, the system start 2nd recovery from a newer checkpoint. - Modified Checkpoint writer to write checkpoint in a different thread. - Added a check to make sure that compute for InputDStreams gets called only for strictly increasing times. - Changed implementation of slice to call getOrCompute on parent DStream in time-increasing order. - Added testcase to test slice. - Fixed testGroupByKeyAndWindow testcase in JavaAPISuite to verify results with expected output in an order-independent manner.
| * | Removed countByKeyAndWindow on paired DStreams, and added ↵Tathagata Das2013-02-143-55/+53
| | | | | | | | | | | | countByValueAndWindow for all DStreams. Updated both scala and java API and testsuites.
| * | Added filter functionality to reduceByKeyAndWindow with inverse. ↵Tathagata Das2013-02-132-17/+34
| | | | | | | | | | | | Consolidated reduceByKeyAndWindow's many functions into smaller number of functions with optional parameters.
| * | Changed scheduler and file input stream to fix bugs in the driver fault ↵Tathagata Das2013-02-138-348/+136
| | | | | | | | | | | | tolerance. Added MasterFailureTest to rigorously test master fault tolerance with file input stream.
| * | Fixed bugs in FileInputDStream and Scheduler that occasionally failed to ↵Tathagata Das2013-02-101-81/+200
| | | | | | | | | | | | reprocess old files after recovering from master failure. Completely modified spark.streaming.FailureTest to test multiple master failures using file input stream.
| * | Added an initial spark job to ensure worker nodes are initialized.Tathagata Das2013-02-091-1/+1
| | |
| * | Merge branch 'mesos-master' into streamingTathagata Das2013-02-077-6/+6
| |\|
| | * Replace old 'master' term with 'driver'.Stephen Haberman2013-01-256-6/+6
| | |
| | * Move JavaAPISuite into spark.streaming.Stephen Haberman2013-01-212-0/+0
| | |
| * | Merge branch 'mesos-streaming' into streamingTathagata Das2013-02-072-0/+50
| |\ \
| | * \ Merge pull request #373 from Reinvigorate/sm-updateStateByKeyTathagata Das2013-02-072-0/+50
| | |\ \ | | | |/ | | |/| StateDStream changes to give updateStateByKey consistent behavior
| | | * adding updateStateByKey object lifecycle testseanm2013-01-202-0/+50
| | | |
| * | | Fixed checkpoint testcasesTathagata Das2013-01-233-172/+129
| | | |
| * | | Added support for rescheduling unprocessed batches on master failure.Tathagata Das2013-01-231-7/+16
| | | |
| * | | Added support for saving input files of FileInputDStream to graph ↵Tathagata Das2013-01-221-20/+44
| | | | | | | | | | | | | | | | checkpoints. Modified 'file input stream with checkpoint' testcase to test recovery of pre-master-failure input files.
| * | | Refactored DStreamCheckpointData.Tathagata Das2013-01-221-6/+6
| |/ /
* / / actor as receiverPrashant Sharma2013-01-221-0/+68
|/ /
* | Fixed streaming testsuite bugsTathagata Das2013-01-207-6/+24
| |
* | Moving tests to appropriate directoryPatrick Wendell2013-01-172-0/+0
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-19 21:56:58 +0000