aboutsummaryrefslogtreecommitdiff
path: root/examples
Commit message (Collapse)AuthorAgeFilesLines
* Merge remote-tracking branch 'apache/master' into driver-testTathagata Das2014-01-1019-30/+76
|\ | | | | | | | | Conflicts: streaming/src/main/scala/org/apache/spark/streaming/DStreamGraph.scala
| * Merge pull request #363 from pwendell/streaming-logsPatrick Wendell2014-01-0919-30/+76
| |\ | | | | | | | | | | | | | | | | | | | | | Set default logging to WARN for Spark streaming examples. This programatically sets the log level to WARN by default for streaming tests. If the user has already specified a log4j.properties file, the user's file will take precedence over this default.
| | * Minor clean-upPatrick Wendell2014-01-091-1/+1
| | |
| | * Set default logging to WARN for Spark streaming examples.Patrick Wendell2014-01-0919-29/+75
| | | | | | | | | | | | | | | | | | This programatically sets the log level to WARN by default for streaming tests. If the user has already specified a log4j.properties file, the user's file will take precedence over this default.
* | | Updated docs based on Patrick's comments in PR 383.Tathagata Das2014-01-102-12/+40
| | |
* | | Merge branch 'standalone-driver' into driver-testTathagata Das2014-01-0948-191/+272
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala examples/src/main/java/org/apache/spark/streaming/examples/JavaNetworkWordCount.java streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
| * | Merge remote-tracking branch 'apache-github/master' into standalone-driverPatrick Wendell2014-01-0828-138/+208
| |\| | | | | | | | | | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/deploy/JsonProtocolSuite.scala pom.xml
| | * Merge pull request #313 from tdas/project-refactorPatrick Wendell2014-01-0710-41/+58
| | |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactored the streaming project to separate external libraries like Twitter, Kafka, Flume, etc. At a high level, these are the following changes. 1. All the external code was put in `SPARK_HOME/external/` as separate SBT projects and Maven modules. Their artifact names are `spark-streaming-twitter`, `spark-streaming-kafka`, etc. Both SparkBuild.scala and pom.xml files have been updated. References to external libraries and repositories have been removed from the settings of root and streaming projects/modules. 2. To avail the external functionality (say, creating a Twitter stream), the developer has to `import org.apache.spark.streaming.twitter._` . For Scala API, the developer has to call `TwitterUtils.createStream(streamingContext, ...)`. For the Java API, the developer has to call `TwitterUtils.createStream(javaStreamingContext, ...)`. 3. Each external project has its own scala and java unit tests. Note the unit tests of each external library use classes of the streaming unit tests (`TestSuiteBase`, `LocalJavaStreamingContext`, etc.). To enable this code sharing among test classes, `dependsOn(streaming % "compile->compile,test->test")` was used in the SparkBuild.scala . In the streaming/pom.xml, an additional `maven-jar-plugin` was necessary to capture this dependency (see comment inside the pom.xml for more information). 4. Jars of the external projects have been added to examples project but not to the assembly project. 5. In some files, imports have been rearrange to conform to the Spark coding guidelines.
| | | * Fixed examples/pom.xml and run-example based on Patrick's suggestions.Tathagata Das2014-01-071-6/+1
| | | |
| | | * Removed XYZFunctions and added XYZUtils as a common Scala and Java interface ↵Tathagata Das2014-01-079-17/+14
| | | | | | | | | | | | | | | | for creating XYZ streams.
| | | * Merge remote-tracking branch 'apache/master' into project-refactorTathagata Das2014-01-0647-66/+77
| | | |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: examples/src/main/java/org/apache/spark/streaming/examples/JavaFlumeEventCount.java streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
| | | * | Changed JavaStreamingContextWith*** to ***Function in streaming.api.java.*** ↵Tathagata Das2014-01-062-6/+6
| | | | | | | | | | | | | | | | | | | | package. Also fixed packages of Flume and MQTT tests.
| | | * | Added pom.xml for external projects and removed unnecessary dependencies and ↵Tathagata Das2013-12-311-15/+30
| | | | | | | | | | | | | | | | | | | | repositoris from other poms and sbt.
| | | * | Refactored kafka, flume, zeromq, mqtt as separate external projects, with ↵Tathagata Das2013-12-306-13/+20
| | | | | | | | | | | | | | | | | | | | their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
| | | * | Refactored streaming project to separate out the twitter functionality.Tathagata Das2013-12-263-1/+4
| | | | |
| | * | | Add log4j exclusion rule to maven.Patrick Wendell2014-01-071-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To make this work I had to rename the defaults file. Otherwise maven's pattern matching rules included it when trying to match other log4j.properties files. I also fixed a bug in the existing maven build where two <transformers> tags were present in assembly/pom.xml such that one overwrote the other.
| | * | | Merge pull request #318 from srowen/masterReynold Xin2014-01-0714-85/+135
| | |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Suggested small changes to Java code for slightly more standard style, encapsulation and in some cases performance Sorry if this is too abrupt or not a welcome set of changes, but thought I'd see if I could contribute a little. I'm a Java developer and just getting seriously into Spark. So I thought I'd suggest a number of small changes to the couple Java parts of the code to make it a little tighter, more standard and even a bit faster. Feel free to take all, some or none of this. Happy to explain any of it.
| | | * | | Issue #318 : minor style updates per review from Reynold XinSean Owen2014-01-0710-33/+2
| | | | | |
| | | * | | Merge remote-tracking branch 'upstream/master'Sean Owen2014-01-0645-59/+63
| | | |\ \ \ | | | | | |/ | | | | |/|
| | | * | | Suggested small changes to Java code for slightly more standard style, ↵Sean Owen2014-01-0214-83/+164
| | | | | | | | | | | | | | | | | | | | | | | | encapsulation and in some cases performance
| | * | | | spark -> org.apache.sparkprabeesh2014-01-078-12/+12
| | | |/ / | | |/| |
| * | | | Merge remote-tracking branch 'apache-github/master' into standalone-driverPatrick Wendell2014-01-0647-67/+78
| |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala core/src/main/scala/org/apache/spark/deploy/client/TestClient.scala core/src/main/scala/org/apache/spark/deploy/master/Master.scala core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
| | * | | Removing SPARK_EXAMPLES_JAR in the codePatrick Wendell2014-01-0544-44/+48
| | | | |
| | * | | run-example -> bin/run-examplePrashant Sharma2014-01-0210-15/+15
| | |/ /
| | * | Merge remote-tracking branch 'origin/master' into conf2Matei Zaharia2013-12-291-1/+1
| | |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala core/src/main/scala/org/apache/spark/scheduler/local/LocalScheduler.scala core/src/main/scala/org/apache/spark/util/MetadataCleaner.scala core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala new-yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala streaming/src/test/scala/org/apache/spark/streaming/WindowOperationsSuite.scala
| | | * Fixed job name in the java streaming example.azuryyu2013-12-241-1/+1
| | | |
| | * | Various fixes to configuration codeMatei Zaharia2013-12-282-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Got rid of global SparkContext.globalConf - Pass SparkConf to serializers and compression codecs - Made SparkConf public instead of private[spark] - Improved API of SparkContext and SparkConf - Switched executor environment vars to be passed through SparkConf - Fixed some places that were still using system properties - Fixed some tests, though others are still failing This still fails several tests in core, repl and streaming, likely due to properties not being set or cleared correctly (some of the tests run fine in isolation).
| | * | spark-544, introducing SparkConf and related configuration overhaul.Prashant Sharma2013-12-253-7/+14
| | |/
* | | Changed the way StreamingContext finds and reads checkpoint files, and added ↵Tathagata Das2014-01-093-10/+42
| | | | | | | | | | | | JavaStreamingContext.getOrCreate.
* | | Added StreamingContext.getOrCreate to for automatic recovery, and added ↵Tathagata Das2014-01-021-0/+58
|/ / | | | | | | RecoverableNetworkWordCount example to use it.
* | Minor style clean-upPatrick Wendell2013-12-251-0/+1
| |
* | Adding better option parsingPatrick Wendell2013-12-251-0/+45
|/
* Use scala.binary.version in POMsMark Hamstra2013-12-151-10/+10
|
* Fix maven build issues in 2.10 branchPatrick Wendell2013-12-131-1/+1
|
* Style fixes and addressed review comments at #221Prashant Sharma2013-12-101-10/+10
|
* Incorporated Patrick's feedback comment on #211 and made maven ↵Prashant Sharma2013-12-071-1/+1
| | | | build/dep-resolution atleast a bit faster.
* Merge branch 'master' of github.com:apache/incubator-spark into scala-2.10-tempPrashant Sharma2013-11-216-16/+19
|\ | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/util/collection/PrimitiveVector.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
| * Merge branch 'master' into removesemicolonscalaHenry Saputra2013-11-192-11/+14
| |\
| | * Enable the Broadcast examples to work in a cluster settingAaron Davidson2013-11-182-11/+14
| | | | | | | | | | | | | | | Since they rely on println to display results, we need to first collect those results to the driver to have them actually display locally.
| * | Remove the semicolons at the end of Scala code to make it more pure Scala code.Henry Saputra2013-11-194-5/+5
| |/ | | | | | | | | | | | | Also remove unused imports as I found them along the way. Remove return statements when returning value in the Scala code. Passing compile and tests.
* | Merge branch 'scala210-master' of github.com:colorant/incubator-spark into ↵Prashant Sharma2013-11-219-30/+274
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | scala-2.10 Conflicts: core/src/main/scala/org/apache/spark/deploy/client/Client.scala core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala
| * | Merge branch 'master' into scala-2.10Raymond Liu2013-11-139-30/+274
| |\|
| | * fix sparkhdfs lr testtgravescs2013-10-291-1/+2
| | |
| | * Makes Spark SIMR ready.Ali Ghodsi2013-10-241-1/+1
| | |
| | * Merge pull request #64 from prabeesh/masterMatei Zaharia2013-10-231-0/+107
| | |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | MQTT Adapter for Spark Streaming MQTT is a machine-to-machine (M2M)/Internet of Things connectivity protocol. It was designed as an extremely lightweight publish/subscribe messaging transport. You may read more about it here http://mqtt.org/ Message Queue Telemetry Transport (MQTT) is an open message protocol for M2M communications. It enables the transfer of telemetry-style data in the form of messages from devices like sensors and actuators, to mobile phones, embedded systems on vehicles, or laptops and full scale computers. The protocol was invented by Andy Stanford-Clark of IBM, and Arlen Nipper of Cirrus Link Solutions This protocol enables a publish/subscribe messaging model in an extremely lightweight way. It is useful for connections with remote locations where line of code and network bandwidth is a constraint. MQTT is one of the widely used protocol for 'Internet of Things'. This protocol is getting much attraction as anything and everything is getting connected to internet and they all produce data. Researchers and companies predict some 25 billion devices will be connected to the internet by 2015. Plugin/Support for MQTT is available in popular MQs like RabbitMQ, ActiveMQ etc. Support for MQTT in Spark will help people with Internet of Things (IoT) projects to use Spark Streaming for their real time data processing needs (from sensors and other embedded devices etc).
| | | * Update MQTTWordCount.scalaPrabeesh K2013-10-221-6/+1
| | | |
| | | * Update MQTTWordCount.scalaPrabeesh K2013-10-221-3/+4
| | | |
| | | * Update MQTTWordCount.scalaPrabeesh K2013-10-181-15/+14
| | | |
| | | * remove unused dependencyprabeesh2013-10-171-5/+0
| | | |
| | | * add maven dependencies for mqttprabeesh2013-10-161-0/+5
| | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-06 07:19:19 +0000