aboutsummaryrefslogtreecommitdiff
path: root/examples
Commit message (Collapse)AuthorAgeFilesLines
...
| | | * | | | | | | | | | Changed JavaStreamingContextWith*** to ***Function in streaming.api.java.*** ↵Tathagata Das2014-01-062-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | package. Also fixed packages of Flume and MQTT tests.
| | | * | | | | | | | | | Added pom.xml for external projects and removed unnecessary dependencies and ↵Tathagata Das2013-12-311-15/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | repositoris from other poms and sbt.
| | | * | | | | | | | | | Refactored kafka, flume, zeromq, mqtt as separate external projects, with ↵Tathagata Das2013-12-306-13/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
| | | * | | | | | | | | | Refactored streaming project to separate out the twitter functionality.Tathagata Das2013-12-263-1/+4
| | | | | | | | | | | | |
| | * | | | | | | | | | | Add log4j exclusion rule to maven.Patrick Wendell2014-01-071-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To make this work I had to rename the defaults file. Otherwise maven's pattern matching rules included it when trying to match other log4j.properties files. I also fixed a bug in the existing maven build where two <transformers> tags were present in assembly/pom.xml such that one overwrote the other.
| | * | | | | | | | | | | Merge pull request #318 from srowen/masterReynold Xin2014-01-0714-85/+135
| | |\ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Suggested small changes to Java code for slightly more standard style, encapsulation and in some cases performance Sorry if this is too abrupt or not a welcome set of changes, but thought I'd see if I could contribute a little. I'm a Java developer and just getting seriously into Spark. So I thought I'd suggest a number of small changes to the couple Java parts of the code to make it a little tighter, more standard and even a bit faster. Feel free to take all, some or none of this. Happy to explain any of it.
| | | * | | | | | | | | | | Issue #318 : minor style updates per review from Reynold XinSean Owen2014-01-0710-33/+2
| | | | | | | | | | | | | |
| | | * | | | | | | | | | | Merge remote-tracking branch 'upstream/master'Sean Owen2014-01-0645-59/+63
| | | |\ \ \ \ \ \ \ \ \ \ \ | | | | | |/ / / / / / / / / | | | | |/| | | | | | | | |
| | | * | | | | | | | | | | Suggested small changes to Java code for slightly more standard style, ↵Sean Owen2014-01-0214-83/+164
| | | | |_|_|_|_|_|_|_|_|/ | | | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | encapsulation and in some cases performance
| | * | | | | | | | | | | spark -> org.apache.sparkprabeesh2014-01-078-12/+12
| | | |/ / / / / / / / / | | |/| | | | | | | | |
| * | | | | | | | | | | Merge remote-tracking branch 'apache-github/master' into standalone-driverPatrick Wendell2014-01-0647-67/+78
| |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala core/src/main/scala/org/apache/spark/deploy/client/TestClient.scala core/src/main/scala/org/apache/spark/deploy/master/Master.scala core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
| | * | | | | | | | | | Removing SPARK_EXAMPLES_JAR in the codePatrick Wendell2014-01-0544-44/+48
| | | | | | | | | | | |
| | * | | | | | | | | | run-example -> bin/run-examplePrashant Sharma2014-01-0210-15/+15
| | |/ / / / / / / / /
| | * | | | | | | | | Merge remote-tracking branch 'origin/master' into conf2Matei Zaharia2013-12-291-1/+1
| | |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala core/src/main/scala/org/apache/spark/scheduler/local/LocalScheduler.scala core/src/main/scala/org/apache/spark/util/MetadataCleaner.scala core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala new-yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala streaming/src/test/scala/org/apache/spark/streaming/WindowOperationsSuite.scala
| | | * | | | | | | | Fixed job name in the java streaming example.azuryyu2013-12-241-1/+1
| | | | | | | | | | |
| | * | | | | | | | | Various fixes to configuration codeMatei Zaharia2013-12-282-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Got rid of global SparkContext.globalConf - Pass SparkConf to serializers and compression codecs - Made SparkConf public instead of private[spark] - Improved API of SparkContext and SparkConf - Switched executor environment vars to be passed through SparkConf - Fixed some places that were still using system properties - Fixed some tests, though others are still failing This still fails several tests in core, repl and streaming, likely due to properties not being set or cleared correctly (some of the tests run fine in isolation).
| | * | | | | | | | | spark-544, introducing SparkConf and related configuration overhaul.Prashant Sharma2013-12-253-7/+14
| | |/ / / / / / / /
* | | | | | | | | | Changed the way StreamingContext finds and reads checkpoint files, and added ↵Tathagata Das2014-01-093-10/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | JavaStreamingContext.getOrCreate.
* | | | | | | | | | Added StreamingContext.getOrCreate to for automatic recovery, and added ↵Tathagata Das2014-01-021-0/+58
|/ / / / / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | RecoverableNetworkWordCount example to use it.
* | | | | | | | | Minor style clean-upPatrick Wendell2013-12-251-0/+1
| | | | | | | | |
* | | | | | | | | Adding better option parsingPatrick Wendell2013-12-251-0/+45
|/ / / / / / / /
* | | | | | | | Use scala.binary.version in POMsMark Hamstra2013-12-151-10/+10
| | | | | | | |
* | | | | | | | Fix maven build issues in 2.10 branchPatrick Wendell2013-12-131-1/+1
| | | | | | | |
* | | | | | | | Style fixes and addressed review comments at #221Prashant Sharma2013-12-101-10/+10
| | | | | | | |
* | | | | | | | Incorporated Patrick's feedback comment on #211 and made maven ↵Prashant Sharma2013-12-071-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | build/dep-resolution atleast a bit faster.
* | | | | | | | Merge branch 'master' of github.com:apache/incubator-spark into scala-2.10-tempPrashant Sharma2013-11-216-16/+19
|\ \ \ \ \ \ \ \ | | |_|_|_|_|_|/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/util/collection/PrimitiveVector.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
| * | | | | | | Merge branch 'master' into removesemicolonscalaHenry Saputra2013-11-192-11/+14
| |\ \ \ \ \ \ \
| | * | | | | | | Enable the Broadcast examples to work in a cluster settingAaron Davidson2013-11-182-11/+14
| | | |_|_|_|_|/ | | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since they rely on println to display results, we need to first collect those results to the driver to have them actually display locally.
| * / | | | | | Remove the semicolons at the end of Scala code to make it more pure Scala code.Henry Saputra2013-11-194-5/+5
| |/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Also remove unused imports as I found them along the way. Remove return statements when returning value in the Scala code. Passing compile and tests.
* | | | | | | Merge branch 'scala210-master' of github.com:colorant/incubator-spark into ↵Prashant Sharma2013-11-219-30/+274
|\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | scala-2.10 Conflicts: core/src/main/scala/org/apache/spark/deploy/client/Client.scala core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala
| * | | | | | | Merge branch 'master' into scala-2.10Raymond Liu2013-11-139-30/+274
| |\| | | | | |
| | * | | | | | fix sparkhdfs lr testtgravescs2013-10-291-1/+2
| | | |_|_|_|/ | | |/| | | |
| | * | | | | Makes Spark SIMR ready.Ali Ghodsi2013-10-241-1/+1
| | | | | | |
| | * | | | | Merge pull request #64 from prabeesh/masterMatei Zaharia2013-10-231-0/+107
| | |\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | MQTT Adapter for Spark Streaming MQTT is a machine-to-machine (M2M)/Internet of Things connectivity protocol. It was designed as an extremely lightweight publish/subscribe messaging transport. You may read more about it here http://mqtt.org/ Message Queue Telemetry Transport (MQTT) is an open message protocol for M2M communications. It enables the transfer of telemetry-style data in the form of messages from devices like sensors and actuators, to mobile phones, embedded systems on vehicles, or laptops and full scale computers. The protocol was invented by Andy Stanford-Clark of IBM, and Arlen Nipper of Cirrus Link Solutions This protocol enables a publish/subscribe messaging model in an extremely lightweight way. It is useful for connections with remote locations where line of code and network bandwidth is a constraint. MQTT is one of the widely used protocol for 'Internet of Things'. This protocol is getting much attraction as anything and everything is getting connected to internet and they all produce data. Researchers and companies predict some 25 billion devices will be connected to the internet by 2015. Plugin/Support for MQTT is available in popular MQs like RabbitMQ, ActiveMQ etc. Support for MQTT in Spark will help people with Internet of Things (IoT) projects to use Spark Streaming for their real time data processing needs (from sensors and other embedded devices etc).
| | | * | | | | Update MQTTWordCount.scalaPrabeesh K2013-10-221-6/+1
| | | | | | | |
| | | * | | | | Update MQTTWordCount.scalaPrabeesh K2013-10-221-3/+4
| | | | | | | |
| | | * | | | | Update MQTTWordCount.scalaPrabeesh K2013-10-181-15/+14
| | | | | | | |
| | | * | | | | remove unused dependencyprabeesh2013-10-171-5/+0
| | | | | | | |
| | | * | | | | add maven dependencies for mqttprabeesh2013-10-161-0/+5
| | | | | | | |
| | | * | | | | added mqtt adapter wordcount exampleprabeesh2013-10-161-0/+112
| | | | | | | |
| | * | | | | | Merge pull request #56 from jerryshao/kafka-0.8-devMatei Zaharia2013-10-213-19/+135
| | |\ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Upgrade Kafka 0.7.2 to Kafka 0.8.0-beta1 for Spark Streaming Conflicts: streaming/pom.xml
| | | * | | | | | Upgrade Kafka 0.7.2 to Kafka 0.8.0-beta1 for Spark Streamingjerryshao2013-10-123-19/+135
| | | | |_|_|/ / | | | |/| | | |
| | * | | | | | Exclusion rules for Maven build files.Reynold Xin2013-10-191-0/+8
| | | | | | | |
| | * | | | | | BroadcastTest2 --> BroadcastTestMosharaf Chowdhury2013-10-162-62/+12
| | | | | | | |
| | * | | | | | Default blockSize is 4MB.Mosharaf Chowdhury2013-10-161-0/+59
| | | |_|_|_|/ | | |/| | | | | | | | | | | | | | | | | | BroadcastTest2 example added for testing broadcasts.
| | * | | | | Fixing spark streaming example and a bug in examples build.Patrick Wendell2013-10-151-4/+9
| | |/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Examples assembly included a log4j.properties which clobbered Spark's - Example had an error where some classes weren't serializable - Did some other clean-up in this example
| | * | | | Remove unnecessary mutable importsNeal Wiggins2013-10-111-2/+0
| | | | | |
* | | | | | Remove deprecated actorFor and use actorSelection everywhere.Prashant Sharma2013-11-121-1/+1
|/ / / / /
* | | | | Merge branch 'scala-2.10' of github.com:ScrapCodes/spark into scala-2.10Prashant Sharma2013-10-101-7/+21
|\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala project/SparkBuild.scala
| * | | | | Merge branch 'master' into wip-merge-masterPrashant Sharma2013-10-081-6/+20
| |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: bagel/pom.xml core/pom.xml core/src/test/scala/org/apache/spark/ui/UISuite.scala examples/pom.xml mllib/pom.xml pom.xml project/SparkBuild.scala repl/pom.xml streaming/pom.xml tools/pom.xml In scala 2.10, a shorter representation is used for naming artifacts so changed to shorter scala version for artifacts and made it a property in pom.
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-29 05:03:29 +0000