aboutsummaryrefslogtreecommitdiff
path: root/streaming/src
Commit message (Collapse)AuthorAgeFilesLines
* Removed extra empty lines.Tathagata Das2013-12-311-1/+0
|
* Removed unnecessary comments.Tathagata Das2013-12-313-55/+8
|
* Refactored kafka, flume, zeromq, mqtt as separate external projects, with ↵Tathagata Das2013-12-3012-978/+81
| | | | their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
* Refactored streaming project to separate out the twitter functionality.Tathagata Das2013-12-263-105/+8
|
* Minor change for PR 277.Tathagata Das2013-12-231-1/+1
|
* Minor formatting fixes.Tathagata Das2013-12-231-5/+4
|
* Added comments to BatchInfo and JobSet, based on Patrick's comment on PR 277.Tathagata Das2013-12-232-3/+26
|
* Minor updated based on comments on PR 277.Tathagata Das2013-12-202-1/+6
|
* Minor changes.Tathagata Das2013-12-189-32/+36
|
* Merge branch 'apache-master' into scheduler-updateTathagata Das2013-12-1841-397/+445
|\ | | | | | | | | | | Conflicts: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/dstream/ForEachDStream.scala
| * Style fixes and addressed review comments at #221Prashant Sharma2013-12-101-2/+2
| |
| * Merge branch 'master' of github.com:apache/incubator-spark into scala-2.10-tempPrashant Sharma2013-11-215-11/+12
| |\ | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/util/collection/PrimitiveVector.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
| * \ Merge branch 'scala210-master' of github.com:colorant/incubator-spark into ↵Prashant Sharma2013-11-2122-257/+1454
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | scala-2.10 Conflicts: core/src/main/scala/org/apache/spark/deploy/client/Client.scala core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala
| | * | Various merge correctionsAaron Davidson2013-11-145-63/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I've diff'd this patch against my own -- since they were both created independently, this means that two sets of eyes have gone over all the merge conflicts that were created, so I'm feeling significantly more confident in the resulting PR. @rxin has looked at the changes to the repl and is resoundingly confident that they are correct.
| | * | Merge branch 'master' into scala-2.10Raymond Liu2013-11-142-7/+80
| | |\ \
| | * \ \ Merge branch 'master' into scala-2.10Raymond Liu2013-11-1321-190/+1375
| | |\ \ \
| * | | | | Remove deprecated actorFor and use actorSelection everywhere.Prashant Sharma2013-11-121-1/+1
| |/ / / /
| * | | | fixed some warningsMartin Weindel2013-10-0515-61/+76
| | | | |
| * | | | Akka 2.2 migrationPrashant Sharma2013-09-226-12/+15
| | | | |
| * | | | Few more fixes to tests broken during mergePrashant Sharma2013-09-101-3/+3
| | | | |
| * | | | Merged with masterPrashant Sharma2013-09-0676-699/+1706
| |\ \ \ \
| * | | | | code formatting, The warning related to scope exit and enter is not worth ↵Prashant Sharma2013-07-161-14/+14
| | | | | | | | | | | | | | | | | | | | | | | | fixing as it only affects debugging scopes and nothing else.
| * | | | | Fixed warning erasure -> runtimeClassPrashant Sharma2013-07-161-1/+1
| | | | | |
| * | | | | Fixed warning ThrowablesPrashant Sharma2013-07-161-1/+1
| | | | | |
| * | | | | Fixed warning ClassManifest -> ClassTagPrashant Sharma2013-07-161-1/+1
| | | | | |
| * | | | | Merge branch 'master' into master-mergePrashant Sharma2013-07-125-18/+25
| |\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: README.md core/pom.xml core/src/main/scala/spark/deploy/JsonProtocol.scala core/src/main/scala/spark/deploy/LocalSparkCluster.scala core/src/main/scala/spark/deploy/master/Master.scala core/src/main/scala/spark/deploy/master/MasterWebUI.scala core/src/main/scala/spark/deploy/worker/Worker.scala core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala core/src/main/scala/spark/storage/BlockManagerUI.scala core/src/main/scala/spark/util/AkkaUtils.scala pom.xml project/SparkBuild.scala streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala
| * \ \ \ \ \ Merge branch 'master' into master-mergePrashant Sharma2013-07-0315-148/+249
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/pom.xml core/src/main/scala/spark/MapOutputTracker.scala core/src/main/scala/spark/RDD.scala core/src/main/scala/spark/RDDCheckpointData.scala core/src/main/scala/spark/SparkContext.scala core/src/main/scala/spark/Utils.scala core/src/main/scala/spark/api/python/PythonRDD.scala core/src/main/scala/spark/deploy/client/Client.scala core/src/main/scala/spark/deploy/master/MasterWebUI.scala core/src/main/scala/spark/deploy/worker/Worker.scala core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala core/src/main/scala/spark/rdd/BlockRDD.scala core/src/main/scala/spark/rdd/ZippedRDD.scala core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala core/src/main/scala/spark/storage/BlockManager.scala core/src/main/scala/spark/storage/BlockManagerMaster.scala core/src/main/scala/spark/storage/BlockManagerMasterActor.scala core/src/main/scala/spark/storage/BlockManagerUI.scala core/src/main/scala/spark/util/AkkaUtils.scala core/src/test/scala/spark/SizeEstimatorSuite.scala pom.xml project/SparkBuild.scala repl/src/main/scala/spark/repl/SparkILoop.scala repl/src/test/scala/spark/repl/ReplSuite.scala streaming/src/main/scala/spark/streaming/StreamingContext.scala streaming/src/main/scala/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala streaming/src/main/scala/spark/streaming/util/MasterFailureTest.scala
| * | | | | | | Fixied other warningsPrashant Sharma2013-04-291-3/+1
| | | | | | | |
| * | | | | | | Fixed warning: erasure -> runtimeClassPrashant Sharma2013-04-291-4/+4
| | | | | | | |
| * | | | | | | Fixed Warning: ClassManifest -> ClassTagPrashant Sharma2013-04-2939-239/+285
| | | | | | | |
| * | | | | | | Fixed breaking tests in streaming checkpoint suite. Changed RichInt to Int ↵Prashant Sharma2013-04-251-16/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | as it is final and not serializable
| * | | | | | | scala 2.10 and master mergePrashant Sharma2013-04-244-15/+16
| | | | | | | |
* | | | | | | | Added StatsReportListener to generate processing time statistics across ↵Tathagata Das2013-12-182-2/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | multiple batches.
* | | | | | | | Refactored streaming scheduler and added listener interface.Tathagata Das2013-12-1222-195/+496
| |_|_|_|_|_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Refactored Scheduler + JobManager to JobGenerator + JobScheduler and added JobSet for cleaner code. Moved scheduler related code to streaming.scheduler package. - Added StreamingListener trait (similar to SparkListener) to enable gathering to streaming stats like processing times and delays. StreamingContext.addListener() to added listeners. - Deduped some code in streaming tests by modifying TestSuiteBase, and added StreamingListenerSuite.
* | | | | | | Another set of changes to remove unnecessary semicolon (;) from Scala code.Henry Saputra2013-11-191-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Passed the sbt/sbt compile and test
* | | | | | | Remove the semicolons at the end of Scala code to make it more pure Scala code.Henry Saputra2013-11-195-10/+9
| |_|_|_|_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Also remove unused imports as I found them along the way. Remove return statements when returning value in the Scala code. Passing compile and tests.
* | | | | | Made block generator thread safe to fix Kafka bug.Tathagata Das2013-11-122-7/+80
| |_|_|_|/ |/| | | |
* | | | | Merge branch 'apache-master' into transformTathagata Das2013-10-258-15/+176
|\ \ \ \ \
| * | | | | Style fixesPatrick Wendell2013-10-241-9/+9
| | | | | |
| * | | | | Spacing fixPatrick Wendell2013-10-241-4/+4
| | | | | |
| * | | | | Small spacing fixPatrick Wendell2013-10-241-2/+2
| | | | | |
| * | | | | Adding Java versions and associated testsPatrick Wendell2013-10-244-0/+68
| | | | | |
| * | | | | Some clean-up of testsPatrick Wendell2013-10-243-7/+10
| | | | | |
| * | | | | Removing Java for nowPatrick Wendell2013-10-241-7/+0
| | | | | |
| * | | | | Adding testsPatrick Wendell2013-10-242-5/+88
| | | | | |
| * | | | | Add a `repartition` operator.Patrick Wendell2013-10-242-0/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds an operator called repartition with more straightforward semantics than the current `coalesce` operator. There are a few use cases where this operator is useful: 1. If a user wants to increase the number of partitions in the RDD. This is more common now with streaming. E.g. a user is ingesting data on one node but they want to add more partitions to ensure parallelism of subsequent operations across threads or the cluster. Right now they have to call rdd.coalesce(numSplits, shuffle=true) - that's super confusing. 2. If a user has input data where the number of partitions is not known. E.g. > sc.textFile("some file").coalesce(50).... This is both vague semantically (am I growing or shrinking this RDD) but also, may not work correctly if the base RDD has fewer than 50 partitions. The new operator forces shuffles every time, so it will always produce exactly the number of new partitions. It also throws an exception rather than silently not-working if a bad input is passed. I am currently adding streaming tests (requires refactoring some of the test suite to allow testing at partition granularity), so this is not ready for merge yet. But feedback is welcome.
* | | | | | Fixed accidental bug.Tathagata Das2013-10-241-1/+1
| | | | | |
* | | | | | Merge branch 'apache-master' into transformTathagata Das2013-10-242-0/+124
|\| | | | |
| * | | | | Merge pull request #64 from prabeesh/masterMatei Zaharia2013-10-232-0/+124
| |\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | MQTT Adapter for Spark Streaming MQTT is a machine-to-machine (M2M)/Internet of Things connectivity protocol. It was designed as an extremely lightweight publish/subscribe messaging transport. You may read more about it here http://mqtt.org/ Message Queue Telemetry Transport (MQTT) is an open message protocol for M2M communications. It enables the transfer of telemetry-style data in the form of messages from devices like sensors and actuators, to mobile phones, embedded systems on vehicles, or laptops and full scale computers. The protocol was invented by Andy Stanford-Clark of IBM, and Arlen Nipper of Cirrus Link Solutions This protocol enables a publish/subscribe messaging model in an extremely lightweight way. It is useful for connections with remote locations where line of code and network bandwidth is a constraint. MQTT is one of the widely used protocol for 'Internet of Things'. This protocol is getting much attraction as anything and everything is getting connected to internet and they all produce data. Researchers and companies predict some 25 billion devices will be connected to the internet by 2015. Plugin/Support for MQTT is available in popular MQs like RabbitMQ, ActiveMQ etc. Support for MQTT in Spark will help people with Internet of Things (IoT) projects to use Spark Streaming for their real time data processing needs (from sensors and other embedded devices etc).
| | * | | | | Update MQTTInputDStream.scalaPrabeesh K2013-10-181-4/+11
| | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-16 11:46:24 +0000