aboutsummaryrefslogtreecommitdiff
path: root/external
Commit message (Collapse)AuthorAgeFilesLines
* SPARK-1121: Include avro for yarn-alpha buildsPatrick Wendell2014-03-025-0/+70
| | | | | | | | | | | | | | | | | This lets us explicitly include Avro based on a profile for 0.23.X builds. It makes me sad how convoluted it is to express this logic in Maven. @tgraves and @sryza curious if this works for you. I'm also considering just reverting to how it was before. The only real problem was that Spark advertised a dependency on Avro even though it only really depends transitively on Avro through other deps. Author: Patrick Wendell <pwendell@gmail.com> Closes #49 from pwendell/avro-build-fix and squashes the following commits: 8d6ee92 [Patrick Wendell] SPARK-1121: Add avro to yarn-alpha profile
* SPARK-1084.2 (resubmitted)Sean Owen2014-03-021-4/+0
| | | | | | | | | | | | | (Ported from https://github.com/apache/incubator-spark/pull/650 ) This adds one more change though, to fix the scala version warning introduced by json4s recently. Author: Sean Owen <sowen@cloudera.com> Closes #32 from srowen/SPARK-1084.2 and squashes the following commits: 9240abd [Sean Owen] Avoid scala version conflict in scalap induced by json4s dependency 1561cec [Sean Owen] Remove "exclude *" dependencies that are causing Maven warnings, and that are apparently unneeded anyway
* Remove remaining references to incubationPatrick Wendell2014-03-025-10/+10
| | | | | | | | | | This removes some loose ends not caught by the other (incubating -> tlp) patches. @markhamstra this updates the version as you mentioned earlier. Author: Patrick Wendell <pwendell@gmail.com> Closes #51 from pwendell/tlp and squashes the following commits: d553b1b [Patrick Wendell] Remove remaining references to incubation
* SPARK-1071: Tidy logging strategy and use of log4jSean Owen2014-02-235-25/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Prompted by a recent thread on the mailing list, I tried and failed to see if Spark can be made independent of log4j. There are a few cases where control of the underlying logging is pretty useful, and to do that, you have to bind to a specific logger. Instead I propose some tidying that leaves Spark's use of log4j, but gets rid of warnings and should still enable downstream users to switch. The idea is to pipe everything (except log4j) through SLF4J, and have Spark use SLF4J directly when logging, and where Spark needs to output info (REPL and tests), bind from SLF4J to log4j. This leaves the same behavior in Spark. It means that downstream users who want to use something except log4j should: - Exclude dependencies on log4j, slf4j-log4j12 from Spark - Include dependency on log4j-over-slf4j - Include dependency on another logger X, and another slf4j-X - Recreate any log config that Spark does, that is needed, in the other logger's config That sounds about right. Here are the key changes: - Include the jcl-over-slf4j shim everywhere by depending on it in core. - Exclude dependencies on commons-logging from third-party libraries. - Include the jul-to-slf4j shim everywhere by depending on it in core. - Exclude slf4j-* dependencies from third-party libraries to prevent collision or warnings - Added missing slf4j-log4j12 binding to GraphX, Bagel module tests And minor/incidental changes: - Update to SLF4J 1.7.5, which happily matches Hadoop 2’s version and is a recommended update over 1.7.2 - (Remove a duplicate HBase dependency declaration in SparkBuild.scala) - (Remove a duplicate mockito dependency declaration that was causing warnings and bugging me) Author: Sean Owen <sowen@cloudera.com> Closes #570 from srowen/SPARK-1071 and squashes the following commits: 52eac9f [Sean Owen] Add slf4j-over-log4j12 dependency to core (non-test) and remove it from things that depend on core. 77a7fa9 [Sean Owen] SPARK-1071: Tidy logging strategy and use of log4j
* Merge pull request #557 from ScrapCodes/style. Closes #557.Patrick Wendell2014-02-094-20/+26
| | | | | | | | | | | | | | | | | | | | | SPARK-1058, Fix Style Errors and Add Scala Style to Spark Build. Author: Patrick Wendell <pwendell@gmail.com> Author: Prashant Sharma <scrapcodes@gmail.com> == Merge branch commits == commit 1a8bd1c059b842cb95cc246aaea74a79fec684f4 Author: Prashant Sharma <scrapcodes@gmail.com> Date: Sun Feb 9 17:39:07 2014 +0530 scala style fixes commit f91709887a8e0b608c5c2b282db19b8a44d53a43 Author: Patrick Wendell <pwendell@gmail.com> Date: Fri Jan 24 11:22:53 2014 -0800 Adding scalastyle snapshot
* Merge pull request #542 from markhamstra/versionBump. Closes #542.Mark Hamstra2014-02-085-5/+5
| | | | | | | | | | | | | | | | | | Version number to 1.0.0-SNAPSHOT Since 0.9.0-incubating is done and out the door, we shouldn't be building 0.9.0-incubating-SNAPSHOT anymore. @pwendell Author: Mark Hamstra <markhamstra@gmail.com> == Merge branch commits == commit 1b00a8a7c1a7f251b4bb3774b84b9e64758eaa71 Author: Mark Hamstra <markhamstra@gmail.com> Date: Wed Feb 5 09:30:32 2014 -0800 Version number to 1.0.0-SNAPSHOT
* Merge pull request #529 from hsaputra/cleanup_right_arrowop_scalaHenry Saputra2014-02-023-9/+6
| | | | | | | | | | | | | | | | | | | | | Change the ⇒ character (maybe from scalariform) to => in Scala code for style consistency Looks like there are some ⇒ Unicode character (maybe from scalariform) in Scala code. This PR is to change it to => to get some consistency on the Scala code. If we want to use ⇒ as default we could use sbt plugin scalariform to make sure all Scala code has ⇒ instead of => And remove unused imports found in TwitterInputDStream.scala while I was there =) Author: Henry Saputra <hsaputra@apache.org> == Merge branch commits == commit 29c1771d346dff901b0b778f764e6b4409900234 Author: Henry Saputra <hsaputra@apache.org> Date: Sat Feb 1 22:05:16 2014 -0800 Change the ⇒ character (maybe from scalariform) to => in Scala code for style consistency.
* Changed SparkConf to not be serializable. And also fixed unit-test log paths ↵Tathagata Das2014-01-149-5/+9
| | | | in log4j.properties of external modules.
* Removed StreamingContext.registerInputStream and registerOutputStream - they ↵Tathagata Das2014-01-135-11/+4
| | | | were useless as InputDStream has been made to register itself. Also made DStream.register() private[streaming] - not useful to expose the confusing function. Updated a lot of documentation.
* Fixed import formatting.Tathagata Das2014-01-125-5/+5
|
* Fixed persistence logic of WindowedDStream, and fixed default persistence ↵Tathagata Das2014-01-124-1/+8
| | | | level of input streams.
* Moved DStream, DStreamCheckpointData and PairDStream from ↵Tathagata Das2014-01-125-5/+10
| | | | org.apache.spark.streaming to org.apache.spark.streaming.dstream.
* Fixed bugs.Tathagata Das2014-01-122-2/+2
|
* Merge pull request #373 from jerryshao/kafka-upgradePatrick Wendell2014-01-111-2/+2
|\ | | | | | | Upgrade Kafka dependecy to 0.8.0 release version
| * Upgrade Kafka dependecy to 0.8.0 release versionjerryshao2014-01-101-2/+2
| |
* | Change clientId to random clientId Prabeesh K2014-01-101-1/+1
|/ | | Returns a randomly generated client identifier based on the current user's login name and the system time.
* Removed XYZFunctions and added XYZUtils as a common Scala and Java interface ↵Tathagata Das2014-01-0726-629/+369
| | | | for creating XYZ streams.
* Merge remote-tracking branch 'apache/master' into project-refactorTathagata Das2014-01-061-1/+1
| | | | | | | | | | Conflicts: examples/src/main/java/org/apache/spark/streaming/examples/JavaFlumeEventCount.java streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
* Changed JavaStreamingContextWith*** to ***Function in streaming.api.java.*** ↵Tathagata Das2014-01-0613-70/+61
| | | | package. Also fixed packages of Flume and MQTT tests.
* Removed extra empty lines.Tathagata Das2013-12-312-2/+0
|
* Added pom.xml for external projects and removed unnecessary dependencies and ↵Tathagata Das2013-12-315-0/+476
| | | | repositoris from other poms and sbt.
* Refactored kafka, flume, zeromq, mqtt as separate external projects, with ↵Tathagata Das2013-12-3035-30/+1894
| | | | their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
* Refactored streaming project to separate out the twitter functionality.Tathagata Das2013-12-263-0/+135
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-26 12:49:25 +0000