aboutsummaryrefslogtreecommitdiff
path: root/project
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #557 from ScrapCodes/style. Closes #557.Patrick Wendell2014-02-094-7/+12
| | | | | | | | | | | | | | | | | | | | | SPARK-1058, Fix Style Errors and Add Scala Style to Spark Build. Author: Patrick Wendell <pwendell@gmail.com> Author: Prashant Sharma <scrapcodes@gmail.com> == Merge branch commits == commit 1a8bd1c059b842cb95cc246aaea74a79fec684f4 Author: Prashant Sharma <scrapcodes@gmail.com> Date: Sun Feb 9 17:39:07 2014 +0530 scala style fixes commit f91709887a8e0b608c5c2b282db19b8a44d53a43 Author: Patrick Wendell <pwendell@gmail.com> Date: Fri Jan 24 11:22:53 2014 -0800 Adding scalastyle snapshot
* Merge pull request #542 from markhamstra/versionBump. Closes #542.Mark Hamstra2014-02-081-1/+1
| | | | | | | | | | | | | | | | | | Version number to 1.0.0-SNAPSHOT Since 0.9.0-incubating is done and out the door, we shouldn't be building 0.9.0-incubating-SNAPSHOT anymore. @pwendell Author: Mark Hamstra <markhamstra@gmail.com> == Merge branch commits == commit 1b00a8a7c1a7f251b4bb3774b84b9e64758eaa71 Author: Mark Hamstra <markhamstra@gmail.com> Date: Wed Feb 5 09:30:32 2014 -0800 Version number to 1.0.0-SNAPSHOT
* modified SparkPluginBuild.scala to use https protocol for accessing github.sarutak2014-01-271-1/+1
|
* Increase JUnit test verbosity under SBT.Josh Rosen2014-01-251-1/+2
| | | | | | | | | Upgrade junit-interface plugin from 0.9 to 0.10. I noticed that the JavaAPISuite tests didn't appear to display any output locally or under Jenkins, making it difficult to know whether they were running. This change increases the verbosity to more closely match the ScalaTest tests.
* Add jblas dependencyJianping J Wang2014-01-231-1/+1
|
* Also add graphx commons-math3 dependeny in sbt buildSean Owen2014-01-221-1/+4
|
* Merge pull request #462 from mateiz/conf-file-fixPatrick Wendell2014-01-181-1/+0
| | | | | | | | | | | | | Remove Typesafe Config usage and conf files to fix nested property names With Typesafe Config we had the subtle problem of no longer allowing nested property names, which are used for a few of our properties: http://apache-spark-developers-list.1001551.n3.nabble.com/Config-properties-broken-in-master-td208.html This PR is for branch 0.9 but should be added into master too. (cherry picked from commit 34e911ce9a9f91f3259189861779032069257852) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* Merge pull request #367 from ankurdave/graphxPatrick Wendell2014-01-131-4/+10
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | GraphX: Unifying Graphs and Tables GraphX extends Spark's distributed fault-tolerant collections API and interactive console with a new graph API which leverages recent advances in graph systems (e.g., [GraphLab](http://graphlab.org)) to enable users to easily and interactively build, transform, and reason about graph structured data at scale. See http://amplab.github.io/graphx/. Thanks to @jegonzal, @rxin, @ankurdave, @dcrankshaw, @jianpingjwang, @amatsukawa, @kellrott, and @adamnovak. Tasks left: - [x] Graph-level uncache - [x] Uncache previous iterations in Pregel - [x] ~~Uncache previous iterations in GraphLab~~ (postponed to post-release) - [x] - Describe GC issue with GraphLab - [ ] Write `docs/graphx-programming-guide.md` - [x] - Mention future Bagel support in docs - [ ] - Section on caching/uncaching in docs: As with Spark, cache something that is used more than once. In an iterative algorithm, try to cache and force (i.e., materialize) something every iteration, then uncache the cached things that depended on the newly materialized RDD but that won't be referenced again. - [x] Undo modifications to core collections and instead copy them to org.apache.spark.graphx - [x] Make Graph serializable to work around capture in Spark shell - [x] Rename graph -> graphx in package name and subproject - [x] Remove standalone PageRank - [x] ~~Fix amplab/graphx#52 by checking `iter.hasNext`~~
| * Merge branch 'master' into graphxReynold Xin2014-01-131-1/+2
| |\
| * | graph -> graphxAnkur Dave2014-01-091-6/+6
| | |
| * | Merge remote-tracking branch 'spark-upstream/master' into HEADAnkur Dave2014-01-082-89/+173
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: README.md core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala core/src/main/scala/org/apache/spark/util/collection/PrimitiveKeyOpenHashMap.scala pom.xml project/SparkBuild.scala repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
| * \ \ Merge branch 'master' of github.com:apache/incubator-sparkReynold Xin2013-11-252-4/+6
| |\ \ \
| * \ \ \ Merge remote-tracking branch 'spark-upstream/master'Ankur Dave2013-10-301-7/+22
| |\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: project/SparkBuild.scala
| * \ \ \ \ Merge branch 'master' of https://github.com/apache/incubator-spark into ↵Joseph E. Gonzalez2013-10-181-0/+1
| |\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | indexedrdd_graphx
| * \ \ \ \ \ merged with upstream changesJoseph E. Gonzalez2013-10-141-7/+14
| |\ \ \ \ \ \
| * | | | | | | GraphX now builds with all merged changes.Joseph E. Gonzalez2013-09-171-7/+9
| | | | | | | |
| * | | | | | | Merging latest changes from spark main branchJoseph E. Gonzalez2013-09-174-74/+123
| |\ \ \ \ \ \ \
| * \ \ \ \ \ \ \ Merged graphx from @rxin into masterJoseph E. Gonzalez2013-08-061-1/+5
| |\ \ \ \ \ \ \ \
| | * \ \ \ \ \ \ \ Merge branch 'master' of github.com:mesos/spark into graphReynold Xin2013-06-292-7/+26
| | |\ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: run run2.cmd
| | * \ \ \ \ \ \ \ \ Merge branch 'master' of github.com:mesos/spark into graphReynold Xin2013-06-011-2/+3
| | |\ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: run
| | * \ \ \ \ \ \ \ \ \ Merge branch 'master' of github.com:mesos/spark into graphReynold Xin2013-05-023-27/+71
| | |\ \ \ \ \ \ \ \ \ \
| | * | | | | | | | | | | Code to run bagel vs graph experiments.Reynold Xin2013-04-071-1/+1
| | | | | | | | | | | | |
| | * | | | | | | | | | | Merge branch 'master' of github.com:mesos/spark into graphReynold Xin2013-04-011-1/+1
| | |\ \ \ \ \ \ \ \ \ \ \
| | * \ \ \ \ \ \ \ \ \ \ \ Merge branch 'master' into graphReynold Xin2013-03-181-9/+11
| | |\ \ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: run2.cmd
| | * \ \ \ \ \ \ \ \ \ \ \ \ Merge branch 'master' into graphReynold Xin2013-02-191-1/+2
| | |\ \ \ \ \ \ \ \ \ \ \ \ \
| | * | | | | | | | | | | | | | Maven and sbt build changes for SparkGraph.Reynold Xin2013-02-191-1/+5
| | | | | | | | | | | | | | | |
* | | | | | | | | | | | | | | | Adjusted visibility of various components.Reynold Xin2014-01-131-0/+7
| |_|_|_|_|_|_|_|_|_|_|_|_|_|/ |/| | | | | | | | | | | | | |
* | | | | | | | | | | | | | | Merge pull request #373 from jerryshao/kafka-upgradePatrick Wendell2014-01-111-9/+9
|\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Upgrade Kafka dependecy to 0.8.0 release version
| * | | | | | | | | | | | | | | Upgrade Kafka dependecy to 0.8.0 release versionjerryshao2014-01-101-9/+9
| | |_|_|_|_|_|_|_|_|_|_|_|_|/ | |/| | | | | | | | | | | | |
* | | | | | | | | | | | | | | Merge remote-tracking branch 'apache-github/master' into standalone-driverPatrick Wendell2014-01-082-27/+67
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/deploy/JsonProtocolSuite.scala pom.xml
| * | | | | | | | | | | | | | Merge pull request #313 from tdas/project-refactorPatrick Wendell2014-01-071-26/+67
| |\ \ \ \ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactored the streaming project to separate external libraries like Twitter, Kafka, Flume, etc. At a high level, these are the following changes. 1. All the external code was put in `SPARK_HOME/external/` as separate SBT projects and Maven modules. Their artifact names are `spark-streaming-twitter`, `spark-streaming-kafka`, etc. Both SparkBuild.scala and pom.xml files have been updated. References to external libraries and repositories have been removed from the settings of root and streaming projects/modules. 2. To avail the external functionality (say, creating a Twitter stream), the developer has to `import org.apache.spark.streaming.twitter._` . For Scala API, the developer has to call `TwitterUtils.createStream(streamingContext, ...)`. For the Java API, the developer has to call `TwitterUtils.createStream(javaStreamingContext, ...)`. 3. Each external project has its own scala and java unit tests. Note the unit tests of each external library use classes of the streaming unit tests (`TestSuiteBase`, `LocalJavaStreamingContext`, etc.). To enable this code sharing among test classes, `dependsOn(streaming % "compile->compile,test->test")` was used in the SparkBuild.scala . In the streaming/pom.xml, an additional `maven-jar-plugin` was necessary to capture this dependency (see comment inside the pom.xml for more information). 4. Jars of the external projects have been added to examples project but not to the assembly project. 5. In some files, imports have been rearrange to conform to the Spark coding guidelines.
| | * \ \ \ \ \ \ \ \ \ \ \ \ \ Merge remote-tracking branch 'apache/master' into project-refactorTathagata Das2014-01-061-14/+40
| | |\ \ \ \ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: examples/src/main/java/org/apache/spark/streaming/examples/JavaFlumeEventCount.java streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
| | * | | | | | | | | | | | | | | Added pom.xml for external projects and removed unnecessary dependencies and ↵Tathagata Das2013-12-311-14/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | repositoris from other poms and sbt.
| | * | | | | | | | | | | | | | | Refactored kafka, flume, zeromq, mqtt as separate external projects, with ↵Tathagata Das2013-12-301-25/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
| | * | | | | | | | | | | | | | | Refactored streaming project to separate out the twitter functionality.Tathagata Das2013-12-261-2/+11
| | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | Merge pull request #331 from holdenk/masterReynold Xin2014-01-071-1/+0
| |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a script to download sbt if not present on the system As per the discussion on the dev mailing list this script will use the system sbt if present or otherwise attempt to install the sbt launcher. The fall back error message in the event it fails instructs the user to install sbt. While the URLs it fetches from aren't controlled by the spark project directly, they are stable and the current authoritative sources.
| | * | | | | | | | | | | | | | | | Use awk to extract the versionHolden Karau2014-01-061-3/+0
| | | | | | | | | | | | | | | | | |
| | * | | | | | | | | | | | | | | | CR feedback (sbt -> sbt/sbt and correct JAR path in script) :)Holden Karau2014-01-051-1/+1
| | | | | | | | | | | | | | | | | |
| | * | | | | | | | | | | | | | | | Add a script to download sbt if not present on the systemHolden Karau2014-01-041-0/+2
| | | |/ / / / / / / / / / / / / / | | |/| | | | | | | | | | | | | |
* | / | | | | | | | | | | | | | | Adding unit tests and some refactoring to promote testability.Patrick Wendell2014-01-071-0/+1
|/ / / / / / / / / / / / / / / /
* | | | | | | | | | | | | | | | Merge pull request #340 from ScrapCodes/sbt-fixesPatrick Wendell2014-01-061-5/+3
|\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Made java options to be applied during tests so that they become self explanatory.
| * | | | | | | | | | | | | | | | Made java options to be applied during tests so that they become self ↵Prashant Sharma2014-01-061-5/+3
| |/ / / / / / / / / / / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | explanatory.
* / / / / / / / / / / / / / / / SPARK-1005 Ning upgradePrashant Sharma2014-01-061-1/+1
|/ / / / / / / / / / / / / / /
* | | | | | | | | | | | | | | Merge remote-tracking branch 'apache-github/master' into remove-binariesPatrick Wendell2014-01-031-7/+25
|\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/DriverSuite.scala docs/python-programming-guide.md
| * | | | | | | | | | | | | | | Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2Raymond Liu2014-01-031-8/+8
| | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | Add yarn/common/src/test dir in building scriptRaymond Liu2014-01-031-0/+7
| | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | Use unmanaged source dir to include common yarn codeRaymond Liu2014-01-031-11/+15
| | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | Reorganize yarn related codes into sub projects to remove duplicate files.Raymond Liu2014-01-031-8/+15
| | | | | | | | | | | | | | | |
* | | | | | | | | | | | | | | | Changes on top of Prashant's patch.Patrick Wendell2014-01-031-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Closes #316
* | | | | | | | | | | | | | | | fixed review commentsPrashant Sharma2014-01-031-5/+9
| | | | | | | | | | | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-11 03:09:44 +0000