Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Update Maven build to create assemblies expected by new scripts | Matei Zaharia | 2013-08-29 | 1 | -6/+56 |
| | | | | | | | | | | | | | | | This includes the following changes: - The "assembly" package now builds in Maven by default, and creates an assembly containing both hadoop-client and Spark, unlike the old BigTop distribution assembly that skipped hadoop-client - There is now a bigtop-dist package to build the old BigTop assembly - The repl-bin package is no longer built by default since the scripts don't reply on it; instead it can be enabled with -Prepl-bin - Py4J is now included in the assembly/lib folder as a local Maven repo, so that the Maven package can link to it - run-example now adds the original Spark classpath as well because the Maven examples assembly lists spark-core and such as provided - The various Maven projects add a spark-yarn dependency correctly | ||||
* | Fix finding of assembly JAR, as well as some pointers to ./run | Matei Zaharia | 2013-08-29 | 8 | -13/+13 |
| | |||||
* | Change build and run instructions to use assemblies | Matei Zaharia | 2013-08-29 | 4 | -0/+452 |
| | | | | | | | | | | | | | | | | This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel. | ||||
* | Remove redundant dependencies from POMs | Jey Kottalam | 2013-08-18 | 1 | -4/+0 |
| | |||||
* | Updates to repl and example POMs to match SBT build | Jey Kottalam | 2013-08-16 | 1 | -0/+10 |
| | |||||
* | Maven build now also works with YARN | Jey Kottalam | 2013-08-16 | 1 | -57/+0 |
| | |||||
* | Don't mark hadoop-client as 'provided' | Jey Kottalam | 2013-08-16 | 1 | -1/+0 |
| | |||||
* | Maven build now works with CDH hadoop-2.0.0-mr1 | Jey Kottalam | 2013-08-16 | 1 | -44/+0 |
| | |||||
* | Initial changes to make Maven build agnostic of hadoop version | Jey Kottalam | 2013-08-16 | 1 | -84/+60 |
| | |||||
* | make SparkHadoopUtil a member of SparkEnv | Jey Kottalam | 2013-08-15 | 1 | -2/+1 |
| | |||||
* | Merge pull request #762 from shivaram/sgd-cleanup | Evan Sparks | 2013-08-11 | 1 | -0/+85 |
|\ | | | | | Refactor SGD options into a new class. | ||||
| * | Add setters for optimizer, gradient in SGD. | Shivaram Venkataraman | 2013-08-08 | 1 | -1/+1 |
| | | | | | | | | Also remove java-specific constructor for LabeledPoint. | ||||
| * | Merge branch 'master' of git://github.com/mesos/spark into sgd-cleanup | Shivaram Venkataraman | 2013-08-06 | 1 | -0/+116 |
| |\ | | | | | | | | | | | | | Conflicts: mllib/src/main/scala/spark/mllib/util/MLUtils.scala | ||||
| * | | Refactor GLM algorithms and add Java tests | Shivaram Venkataraman | 2013-08-06 | 1 | -0/+85 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This change adds Java examples and unit tests for all GLM algorithms to make sure the MLLib interface works from Java. Changes include - Introduce LabeledPoint and avoid using Doubles in train arguments - Rename train to run in class methods - Make the optimizer a member variable of GLM to make sure the builder pattern works | ||||
* | | | Fixed path to JavaALS.java and JavaKMeans.java, fixed hadoop2-yarn profile | Alexander Pivovarov | 2013-08-10 | 3 | -0/+6 |
| | | | |||||
* | | | Optimize Scala PageRank to use reduceByKey | Matei Zaharia | 2013-08-10 | 1 | -8/+4 |
| | | | |||||
* | | | Merge pull request #789 from MLnick/master | Matei Zaharia | 2013-08-10 | 1 | -0/+50 |
|\ \ \ | | | | | | | | | Adding Scala version of PageRank example | ||||
| * | | | Style changes as per Matei's comments | Nick Pentreath | 2013-08-08 | 1 | -9/+8 |
| | | | | |||||
| * | | | Adding Scala version of PageRank example | Nick Pentreath | 2013-08-07 | 1 | -0/+51 |
| | |/ | |/| | |||||
* | | | Merge pull request #786 from shivaram/mllib-java | Matei Zaharia | 2013-08-09 | 3 | -0/+180 |
|\ \ \ | | | | | | | | | Java fixes, tests and examples for ALS, KMeans | ||||
| * | | | Remove Java-specific constructor for Rating. | Shivaram Venkataraman | 2013-08-08 | 1 | -3/+3 |
| | | | | | | | | | | | | | | | | | | | | The scala constructor works for native type java types. Modify examples to match this. | ||||
| * | | | Java examples, tests for KMeans and ALS | Shivaram Venkataraman | 2013-08-06 | 3 | -0/+180 |
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Changes ALS to accept RDD[Rating] instead of (Int, Int, Double) making it easier to call from Java - Renames class methods from `train` to `run` to enable static methods to be called from Java. - Add unit tests which check if both static / class methods can be called. - Also add examples which port the main() function in ALS, KMeans to the examples project. Couple of minor changes to existing code: - Add a toJavaRDD method in RDD to convert scala RDD to java RDD easily - Workaround a bug where using double[] from Java leads to class cast exception in KMeans init | ||||
* / / | Optimize JavaPageRank to use reduceByKey instead of groupByKey | Matei Zaharia | 2013-08-08 | 1 | -9/+8 |
|/ / | |||||
* | | Got rid of unnecessary map function | stayhf | 2013-08-06 | 1 | -6/+2 |
| | | |||||
* | | changes as reviewer requested | stayhf | 2013-08-06 | 1 | -10/+1 |
| | | |||||
* | | Updated code with reviewer's suggestions | stayhf | 2013-08-05 | 1 | -47/+47 |
| | | |||||
* | | Simple PageRank algorithm implementation in Java for SPARK-760 | stayhf | 2013-08-03 | 1 | -0/+129 |
|/ | |||||
* | Add Apache license headers and LICENSE and NOTICE files | Matei Zaharia | 2013-07-16 | 45 | -1/+766 |
| | |||||
* | pom cleanup | Mark Hamstra | 2013-07-08 | 1 | -1/+0 |
| | |||||
* | Explicit dependencies for scala-library and scalap to prevent 2.9.2 vs. ↵ | Mark Hamstra | 2013-07-08 | 1 | -0/+5 |
| | | | | 2.9.3 problems | ||||
* | Fixing missed hbase dependency in examples hadoop2-yarn profile | Konstantin Boudnik | 2013-07-01 | 1 | -0/+5 |
| | |||||
* | Merge pull request #577 from skumargithub/master | Matei Zaharia | 2013-06-29 | 1 | -0/+50 |
|\ | | | | | Example of cumulative counting using updateStateByKey | ||||
| * | Removed unused code, clarified intent of the program, batch size to 1 second | unknown | 2013-05-06 | 1 | -5/+3 |
| | | |||||
| * | Modified as per TD's suggestions | unknown | 2013-04-30 | 1 | -17/+6 |
| | | |||||
| * | Examaple of cumulative counting using updateStateByKey | unknown | 2013-04-22 | 1 | -0/+63 |
| | | |||||
* | | Merge remote-tracking branch 'mrpotes/master' | Matei Zaharia | 2013-06-29 | 3 | -15/+12 |
|\ \ | |||||
| * | | Fix usage and parameter extraction | James Phillpotts | 2013-06-25 | 3 | -12/+9 |
| | | | |||||
| * | | Include a default OAuth implementation, and update examples and ↵ | James Phillpotts | 2013-06-25 | 3 | -3/+3 |
| | | | | | | | | | | | | JavaStreamingContext | ||||
* | | | Merge branch 'master' into streaming | Tathagata Das | 2013-06-24 | 41 | -167/+825 |
|\| | | | | | | | | | | | | | | Conflicts: .gitignore | ||||
| * | | Merge remote-tracking branch 'milliondreams/casdemo' | Matei Zaharia | 2013-06-18 | 2 | -0/+231 |
| |\ \ | | | | | | | | | | | | | | | | | Conflicts: project/SparkBuild.scala | ||||
| | * | | Fixing the style as per feedback | Rohit Rai | 2013-06-13 | 1 | -35/+37 |
| | | | | |||||
| | * | | Example to write the output to cassandra | Rohit Rai | 2013-06-03 | 1 | -5/+43 |
| | | | | |||||
| | * | | A better way to read column value if you are sure the column exists in every ↵ | Rohit Rai | 2013-06-03 | 1 | -2/+4 |
| | | | | | | | | | | | | | | | | row. | ||||
| | * | | Adding deps to examples/pom.xml | Rohit Rai | 2013-06-02 | 1 | -0/+35 |
| | | | | | | | | | | | | | | | | Fixing exclusion in examples deps in SparkBuild.scala | ||||
| | * | | Removing infix call | Rohit Rai | 2013-06-02 | 1 | -3/+3 |
| | | | | |||||
| | * | | Adding example to make Spark RDD from Cassandra | Rohit Rai | 2013-06-01 | 1 | -0/+154 |
| | | | | |||||
| * | | | Add hBase dependency to examples POM | Ethan Jewett | 2013-05-20 | 1 | -0/+10 |
| | | | | |||||
| * | | | Add hBase example | Ethan Jewett | 2013-05-09 | 1 | -0/+35 |
| | | | | |||||
| * | | | Revert "Merge pull request #596 from esjewett/master" because the | Reynold Xin | 2013-05-09 | 1 | -35/+0 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | dependency on hbase introduces netty-3.2.2 which conflicts with netty-3.5.3 already in Spark. This caused multiple test failures. This reverts commit 0f1b7a06e1f6782711170234f105f1b277e3b04c, reversing changes made to aacca1b8a85bd073ce185a06d6470b070761b2f4. | ||||
| * | | | Switch to using SparkContext method to create RDD | Ethan Jewett | 2013-05-07 | 1 | -2/+2 |
| | | | |