aboutsummaryrefslogtreecommitdiff
path: root/examples
Commit message (Collapse)AuthorAgeFilesLines
...
* | | | | | | | | Merge remote-tracking branch 'spark-upstream/master'Ankur Dave2013-10-306-23/+263
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: project/SparkBuild.scala
| * | | | | | | | Makes Spark SIMR ready.Ali Ghodsi2013-10-241-1/+1
| | | | | | | | |
| * | | | | | | | Merge pull request #64 from prabeesh/masterMatei Zaharia2013-10-231-0/+107
| |\ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | MQTT Adapter for Spark Streaming MQTT is a machine-to-machine (M2M)/Internet of Things connectivity protocol. It was designed as an extremely lightweight publish/subscribe messaging transport. You may read more about it here http://mqtt.org/ Message Queue Telemetry Transport (MQTT) is an open message protocol for M2M communications. It enables the transfer of telemetry-style data in the form of messages from devices like sensors and actuators, to mobile phones, embedded systems on vehicles, or laptops and full scale computers. The protocol was invented by Andy Stanford-Clark of IBM, and Arlen Nipper of Cirrus Link Solutions This protocol enables a publish/subscribe messaging model in an extremely lightweight way. It is useful for connections with remote locations where line of code and network bandwidth is a constraint. MQTT is one of the widely used protocol for 'Internet of Things'. This protocol is getting much attraction as anything and everything is getting connected to internet and they all produce data. Researchers and companies predict some 25 billion devices will be connected to the internet by 2015. Plugin/Support for MQTT is available in popular MQs like RabbitMQ, ActiveMQ etc. Support for MQTT in Spark will help people with Internet of Things (IoT) projects to use Spark Streaming for their real time data processing needs (from sensors and other embedded devices etc).
| | * | | | | | | | Update MQTTWordCount.scalaPrabeesh K2013-10-221-6/+1
| | | | | | | | | |
| | * | | | | | | | Update MQTTWordCount.scalaPrabeesh K2013-10-221-3/+4
| | | | | | | | | |
| | * | | | | | | | Update MQTTWordCount.scalaPrabeesh K2013-10-181-15/+14
| | | | | | | | | |
| | * | | | | | | | remove unused dependencyprabeesh2013-10-171-5/+0
| | | | | | | | | |
| | * | | | | | | | add maven dependencies for mqttprabeesh2013-10-161-0/+5
| | | | | | | | | |
| | * | | | | | | | added mqtt adapter wordcount exampleprabeesh2013-10-161-0/+112
| | | |_|_|/ / / / | | |/| | | | | |
| * | | | | | | | Merge pull request #56 from jerryshao/kafka-0.8-devMatei Zaharia2013-10-213-19/+135
| |\ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Upgrade Kafka 0.7.2 to Kafka 0.8.0-beta1 for Spark Streaming Conflicts: streaming/pom.xml
| | * | | | | | | | Upgrade Kafka 0.7.2 to Kafka 0.8.0-beta1 for Spark Streamingjerryshao2013-10-123-19/+135
| | | | | | | | | |
| * | | | | | | | | Exclusion rules for Maven build files.Reynold Xin2013-10-191-0/+8
| | | | | | | | | |
| * | | | | | | | | BroadcastTest2 --> BroadcastTestMosharaf Chowdhury2013-10-162-62/+12
| | | | | | | | | |
| * | | | | | | | | Default blockSize is 4MB.Mosharaf Chowdhury2013-10-161-0/+59
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | BroadcastTest2 example added for testing broadcasts.
* | | | | | | | | | Merge branch 'master' of https://github.com/apache/incubator-spark into ↵Joseph E. Gonzalez2013-10-181-4/+9
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | indexedrdd_graphx
| * | | | | | | | | Fixing spark streaming example and a bug in examples build.Patrick Wendell2013-10-151-4/+9
| |/ / / / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Examples assembly included a log4j.properties which clobbered Spark's - Example had an error where some classes weren't serializable - Did some other clean-up in this example
* | | | | | | | | merged with upstream changesJoseph E. Gonzalez2013-10-142-11/+23
|\| | | | | | | |
| * | | | | | | | Remove unnecessary mutable importsNeal Wiggins2013-10-111-2/+0
| | |_|_|_|_|_|/ | |/| | | | | |
| * | | | | | | Merging build changes in from 0.8Patrick Wendell2013-10-051-8/+22
| | |_|_|_|_|/ | |/| | | | |
| * | | | | | Update build version in masterPatrick Wendell2013-09-241-1/+1
| | |_|_|_|/ | |/| | | |
* | | | | | Merging latest changes from spark main branchJoseph E. Gonzalez2013-09-1753-353/+1167
|\| | | | |
| * | | | | Minor YARN build cleanupsJey Kottalam2013-09-061-14/+0
| |/ / / /
| * | | | Add missing license headers found with RATMatei Zaharia2013-09-021-0/+17
| | | | |
| * | | | Move some classes to more appropriate packages:Matei Zaharia2013-09-015-13/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * RDD, *RDDFunctions -> org.apache.spark.rdd * Utils, ClosureCleaner, SizeEstimator -> org.apache.spark.util * JavaSerializer, KryoSerializer -> org.apache.spark.serializer
| * | | | Fix some URLsMatei Zaharia2013-09-011-1/+1
| | | | |
| * | | | Initial work to rename package to org.apache.sparkMatei Zaharia2013-09-0153-220/+220
| | | | |
| * | | | Update Maven build to create assemblies expected by new scriptsMatei Zaharia2013-08-291-6/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This includes the following changes: - The "assembly" package now builds in Maven by default, and creates an assembly containing both hadoop-client and Spark, unlike the old BigTop distribution assembly that skipped hadoop-client - There is now a bigtop-dist package to build the old BigTop assembly - The repl-bin package is no longer built by default since the scripts don't reply on it; instead it can be enabled with -Prepl-bin - Py4J is now included in the assembly/lib folder as a local Maven repo, so that the Maven package can link to it - run-example now adds the original Spark classpath as well because the Maven examples assembly lists spark-core and such as provided - The various Maven projects add a spark-yarn dependency correctly
| * | | | Fix finding of assembly JAR, as well as some pointers to ./runMatei Zaharia2013-08-298-13/+13
| | | | |
| * | | | Change build and run instructions to use assembliesMatei Zaharia2013-08-294-0/+452
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
| * | | | Remove redundant dependencies from POMsJey Kottalam2013-08-181-4/+0
| | | | |
| * | | | Updates to repl and example POMs to match SBT buildJey Kottalam2013-08-161-0/+10
| | | | |
| * | | | Maven build now also works with YARNJey Kottalam2013-08-161-57/+0
| | | | |
| * | | | Don't mark hadoop-client as 'provided'Jey Kottalam2013-08-161-1/+0
| | | | |
| * | | | Maven build now works with CDH hadoop-2.0.0-mr1Jey Kottalam2013-08-161-44/+0
| | | | |
| * | | | Initial changes to make Maven build agnostic of hadoop versionJey Kottalam2013-08-161-84/+60
| | | | |
| * | | | make SparkHadoopUtil a member of SparkEnvJey Kottalam2013-08-151-2/+1
| | | | |
| * | | | Merge pull request #762 from shivaram/sgd-cleanupEvan Sparks2013-08-111-0/+85
| |\ \ \ \ | | | | | | | | | | | | Refactor SGD options into a new class.
| | * | | | Add setters for optimizer, gradient in SGD.Shivaram Venkataraman2013-08-081-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | Also remove java-specific constructor for LabeledPoint.
| | * | | | Merge branch 'master' of git://github.com/mesos/spark into sgd-cleanupShivaram Venkataraman2013-08-061-0/+116
| | |\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: mllib/src/main/scala/spark/mllib/util/MLUtils.scala
| | * | | | | Refactor GLM algorithms and add Java testsShivaram Venkataraman2013-08-061-0/+85
| |/ / / / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change adds Java examples and unit tests for all GLM algorithms to make sure the MLLib interface works from Java. Changes include - Introduce LabeledPoint and avoid using Doubles in train arguments - Rename train to run in class methods - Make the optimizer a member variable of GLM to make sure the builder pattern works
| * | | | | Fixed path to JavaALS.java and JavaKMeans.java, fixed hadoop2-yarn profileAlexander Pivovarov2013-08-103-0/+6
| | | | | |
| * | | | | Optimize Scala PageRank to use reduceByKeyMatei Zaharia2013-08-101-8/+4
| | | | | |
| * | | | | Merge pull request #789 from MLnick/masterMatei Zaharia2013-08-101-0/+50
| |\ \ \ \ \ | | | | | | | | | | | | | | Adding Scala version of PageRank example
| | * | | | | Style changes as per Matei's commentsNick Pentreath2013-08-081-9/+8
| | | | | | |
| | * | | | | Adding Scala version of PageRank exampleNick Pentreath2013-08-071-0/+51
| | |/ / / /
| * | | | | Merge pull request #786 from shivaram/mllib-javaMatei Zaharia2013-08-093-0/+180
| |\ \ \ \ \ | | | | | | | | | | | | | | Java fixes, tests and examples for ALS, KMeans
| | * | | | | Remove Java-specific constructor for Rating.Shivaram Venkataraman2013-08-081-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The scala constructor works for native type java types. Modify examples to match this.
| | * | | | | Java examples, tests for KMeans and ALSShivaram Venkataraman2013-08-063-0/+180
| | |/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Changes ALS to accept RDD[Rating] instead of (Int, Int, Double) making it easier to call from Java - Renames class methods from `train` to `run` to enable static methods to be called from Java. - Add unit tests which check if both static / class methods can be called. - Also add examples which port the main() function in ALS, KMeans to the examples project. Couple of minor changes to existing code: - Add a toJavaRDD method in RDD to convert scala RDD to java RDD easily - Workaround a bug where using double[] from Java leads to class cast exception in KMeans init
| * / / / / Optimize JavaPageRank to use reduceByKey instead of groupByKeyMatei Zaharia2013-08-081-9/+8
| |/ / / /
| * | | | Got rid of unnecessary map functionstayhf2013-08-061-6/+2
| | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-16 02:20:28 +0000