| Commit message (Collapse) | Author | Age | Files | Lines | |
|---|---|---|---|---|---|
| * | Merge pull request #313 from tdas/project-refactor | Patrick Wendell | 2014-01-07 | 1 | -6/+1 |
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactored the streaming project to separate external libraries like Twitter, Kafka, Flume, etc. At a high level, these are the following changes. 1. All the external code was put in `SPARK_HOME/external/` as separate SBT projects and Maven modules. Their artifact names are `spark-streaming-twitter`, `spark-streaming-kafka`, etc. Both SparkBuild.scala and pom.xml files have been updated. References to external libraries and repositories have been removed from the settings of root and streaming projects/modules. 2. To avail the external functionality (say, creating a Twitter stream), the developer has to `import org.apache.spark.streaming.twitter._` . For Scala API, the developer has to call `TwitterUtils.createStream(streamingContext, ...)`. For the Java API, the developer has to call `TwitterUtils.createStream(javaStreamingContext, ...)`. 3. Each external project has its own scala and java unit tests. Note the unit tests of each external library use classes of the streaming unit tests (`TestSuiteBase`, `LocalJavaStreamingContext`, etc.). To enable this code sharing among test classes, `dependsOn(streaming % "compile->compile,test->test")` was used in the SparkBuild.scala . In the streaming/pom.xml, an additional `maven-jar-plugin` was necessary to capture this dependency (see comment inside the pom.xml for more information). 4. Jars of the external projects have been added to examples project but not to the assembly project. 5. In some files, imports have been rearrange to conform to the Spark coding guidelines. | ||||
| | * | Fixed examples/pom.xml and run-example based on Patrick's suggestions. | Tathagata Das | 2014-01-07 | 1 | -6/+1 |
| | | | |||||
| * | | CR feedback (sbt -> sbt/sbt and correct JAR path in script) :) | Holden Karau | 2014-01-05 | 1 | -1/+1 |
| |/ | |||||
| * | Merge remote-tracking branch 'apache-github/master' into remove-binaries | Patrick Wendell | 2014-01-03 | 1 | -1/+1 |
| | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/DriverSuite.scala docs/python-programming-guide.md | ||||
| * | sbin/compute-classpath* bin/compute-classpath* | Prashant Sharma | 2014-01-03 | 1 | -1/+1 |
| | | |||||
| * | sbin/spark-class* -> bin/spark-class* | Prashant Sharma | 2014-01-03 | 1 | -1/+1 |
| | | |||||
| * | run-example -> bin/run-example | Prashant Sharma | 2014-01-02 | 1 | -1/+1 |
| | | |||||
| * | Merge branch 'scripts-reorg' of github.com:shane-huang/incubator-spark into ↵ | Prashant Sharma | 2014-01-02 | 1 | -1/+11 |
| | | | | | | | | | | | | | | | | | spark-915-segregate-scripts Conflicts: bin/spark-shell core/pom.xml core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala core/src/main/scala/org/apache/spark/ui/UIWorkloadGenerator.scala core/src/test/scala/org/apache/spark/DriverSuite.scala python/run-tests sbin/compute-classpath.sh sbin/spark-class sbin/stop-slaves.sh | ||||
| * | refactor $FWD variable | Andrew xia | 2013-09-29 | 1 | -1/+1 |
| | | |||||
| * | add scripts in bin | shane-huang | 2013-09-23 | 1 | -2/+2 |
| | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com> | ||||
| * | moved user scripts to bin folder | shane-huang | 2013-09-23 | 1 | -0/+81 |
| Signed-off-by: shane-huang <shengsheng.huang@intel.com> | |||||
 |
index : spark | |
| Mirror of Apache Spark |
| aboutsummaryrefslogtreecommitdiff |