Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Add support for building against hadoop2-yarn : adding new maven profile for it | Mridul Muralidharan | 2013-04-07 | 1 | -0/+37 |
| | |||||
* | Move streaming test initialization into 'before' blocks | Jey Kottalam | 2013-03-28 | 2 | -4/+8 |
| | |||||
* | method first in trait IterableLike is deprecated: use `head' instead | Holden Karau | 2013-03-24 | 1 | -1/+1 |
| | |||||
* | Add a log4j compile dependency to fix build in IntelliJ | Mikhail Bautin | 2013-03-15 | 1 | -1/+1 |
| | | | | | Also rename parent project to spark-parent (otherwise it shows up as "parent" in IntelliJ, which is very confusing). | ||||
* | Forgot equals. | Stephen Haberman | 2013-03-12 | 1 | -1/+1 |
| | |||||
* | More quickly call close in HadoopRDD. | Stephen Haberman | 2013-03-11 | 1 | -35/+9 |
| | | | | | This also refactors out the common "gotNext" iterator pattern into a shared utility class. | ||||
* | Instead of failing to bind to a fixed, already-in-use port, let the OS ↵ | Mark Hamstra | 2013-03-01 | 1 | -8/+10 |
| | | | | choose an available port for TestServer. | ||||
* | bump version to 0.7.1-SNAPSHOT in the subproject poms to keep the maven ↵ | Mark Hamstra | 2013-02-28 | 1 | -1/+1 |
| | | | | build building. | ||||
* | Merge pull request #503 from pwendell/bug-fix | Matei Zaharia | 2013-02-25 | 1 | -1/+1 |
|\ | | | | | createNewSparkContext should use sparkHome/jars/environment. | ||||
| * | createNewSparkContext should use sparkHome/jars/environment. | Patrick Wendell | 2013-02-25 | 1 | -1/+1 |
| | | | | | | | | This fixes a bug introduced by Matei's recent change. | ||||
* | | Pass a code JAR to SparkContext in our examples. Fixes SPARK-594. | Matei Zaharia | 2013-02-25 | 1 | -0/+17 |
|/ | |||||
* | Changed Flume test to use the same port as other tests, so that can be ↵ | Tathagata Das | 2013-02-25 | 1 | -2/+2 |
| | | | | controlled centrally. | ||||
* | Fixed something that was reported as a compile error in ScalaDoc. | Matei Zaharia | 2013-02-25 | 2 | -3/+3 |
| | | | | | | For some reason, ScalaDoc complained about no such constructor for StreamingContext; it doesn't seem like an actual Scala error but it prevented sbt publish and from working because docs weren't built. | ||||
* | Allow passing sparkHome and JARs to StreamingContext constructor | Matei Zaharia | 2013-02-25 | 4 | -10/+68 |
| | | | | | Also warns if spark.cleaner.ttl is not set in the version where you pass your own SparkContext. | ||||
* | Fixed class paths and dependencies based on Matei's comments. | Tathagata Das | 2013-02-24 | 1 | -11/+0 |
| | |||||
* | Added back the initial spark job before starting streaming receivers | Tathagata Das | 2013-02-24 | 1 | -1/+1 |
| | |||||
* | Fixed missing dependencies in streaming/pom.xml | Tathagata Das | 2013-02-24 | 1 | -4/+14 |
| | |||||
* | Updated streaming programming guide with Java API info, and comments from ↵ | Tathagata Das | 2013-02-23 | 2 | -3/+2 |
| | | | | Patrick. | ||||
* | Fixed differences in APIs of StreamingContext and JavaStreamingContext. ↵ | Tathagata Das | 2013-02-23 | 4 | -25/+231 |
| | | | | Change rawNetworkStream to rawSocketStream, and added twitter, actor, zeroMQ streams to JavaStreamingContext. Also added them to JavaAPISuite. | ||||
* | Merge branch 'mesos-streaming' into streaming | Tathagata Das | 2013-02-22 | 3 | -2/+66 |
|\ | |||||
| * | Merge pull request #480 from MLnick/streaming-eg-algebird | Tathagata Das | 2013-02-22 | 1 | -0/+10 |
| |\ | | | | | | | [Streaming] Examples using Twitter's Algebird library | ||||
| | * | Merge remote-tracking branch 'upstream/streaming' into streaming-eg-algebird | Nick Pentreath | 2013-02-21 | 11 | -43/+306 |
| | |\ | |||||
| | * | | Dependencies and refactoring for streaming HLL example, and using ↵ | Nick Pentreath | 2013-02-19 | 1 | -0/+10 |
| | | | | | | | | | | | | | | | | context.twitterStream method | ||||
| * | | | Merge pull request #479 from ScrapCodes/zeromq-streaming | Tathagata Das | 2013-02-22 | 2 | -2/+56 |
| |\ \ \ | | |_|/ | |/| | | Zeromq streaming | ||||
| | * | | fixes corresponding to review feedback at pull request #479 | Prashant Sharma | 2013-02-20 | 1 | -1/+1 |
| | | | | |||||
| | * | | example for demonstrating ZeroMQ stream | Prashant Sharma | 2013-02-19 | 1 | -8/+7 |
| | | | | |||||
| | * | | ZeroMQ stream as receiver | Prashant Sharma | 2013-02-19 | 2 | -0/+55 |
| | |/ | |||||
* | / | Fixed condition in InputDStream isTimeValid. | Tathagata Das | 2013-02-21 | 1 | -1/+1 |
|/ / | |||||
* | | Merge branch 'mesos-streaming' into streaming | Tathagata Das | 2013-02-20 | 6 | -49/+102 |
|\ \ | | | | | | | | | | | | | Conflicts: streaming/src/test/java/spark/streaming/JavaAPISuite.java | ||||
| * | | Small changes that were missing in merge | Patrick Wendell | 2013-02-19 | 2 | -0/+2 |
| | | | |||||
| * | | Use RDD type for `slice` operator in Java. | Patrick Wendell | 2013-02-19 | 1 | -2/+2 |
| | | | | | | | | | | | | | | | | | | This commit uses the RDD type in `slice`, making it available to both normal and pair RDD's in java. It also updates the signature for `slice` to match changes in the Scala API. | ||||
| * | | Use RDD type for `transform` operator in Java. | Patrick Wendell | 2013-02-19 | 2 | -7/+122 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is an improved implementation of the `transform` operator in Java. The main difference is that this allows all four possible types of transform functions 1. JavaRDD -> JavaRDD 2. JavaRDD -> JavaPairRDD 3. JavaPairRDD -> JavaPairRDD 4. JavaPairRDD -> JavaRDD whereas previously only (1) and (3) were possible. Conflicts: streaming/src/test/java/spark/streaming/JavaAPISuite.java | ||||
| * | | Use RDD type for `foreach` operator in Java. | Patrick Wendell | 2013-02-19 | 6 | -11/+21 |
| |/ | |||||
* | | Merge branch 'mesos-master' into streaming | Tathagata Das | 2013-02-20 | 9 | -24/+234 |
|\ \ | |/ |/| | | | | | | | Conflicts: core/src/main/scala/spark/rdd/CheckpointRDD.scala streaming/src/main/scala/spark/streaming/dstream/ReducedWindowedDStream.scala | ||||
| * | Rename "jobs" to "applications" in the standalone cluster | Matei Zaharia | 2013-02-17 | 3 | -9/+9 |
| | | |||||
| * | Make CoGroupedRDDs explicitly have the same key type. | Stephen Haberman | 2013-02-16 | 3 | -3/+3 |
| | | |||||
| * | STREAMING-50: Support transform workaround in JavaPairDStream | Patrick Wendell | 2013-02-12 | 2 | -2/+77 |
| | | | | | | | | | | | | This ports a useful workaround (the `transform` function) to JavaPairDStream. It is necessary to do things like sorting which are not supported yet in the core streaming API. | ||||
| * | Using tuple swap() | Patrick Wendell | 2013-02-11 | 1 | -2/+2 |
| | | |||||
| * | small fix | Patrick Wendell | 2013-02-11 | 1 | -2/+2 |
| | | |||||
| * | Fix for MapPartitions | Patrick Wendell | 2013-02-11 | 2 | -17/+54 |
| | | |||||
| * | Fix for flatmap | Patrick Wendell | 2013-02-11 | 2 | -2/+44 |
| | | |||||
| * | Indentation fix | Patrick Wendell | 2013-02-11 | 1 | -10/+10 |
| | | |||||
| * | Initial cut at replacing K, V in Java files | Patrick Wendell | 2013-02-11 | 2 | -2/+58 |
| | | |||||
* | | Merge branch 'streaming' into ScrapCodes-streaming-actor | Tathagata Das | 2013-02-19 | 46 | -1056/+2026 |
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: docs/plugin-custom-receiver.md streaming/src/main/scala/spark/streaming/StreamingContext.scala streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala streaming/src/main/scala/spark/streaming/dstream/PluggableInputDStream.scala streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala streaming/src/test/scala/spark/streaming/InputStreamsSuite.scala | ||||
| * | | Changed networkStream to socketStream and pluggableNetworkStream to become ↵ | Tathagata Das | 2013-02-18 | 4 | -21/+22 |
| | | | | | | | | | | | | networkStream as a way to create streams from arbitrary network receiver. | ||||
| * | | Merge branch 'streaming' into ScrapCode-streaming | Tathagata Das | 2013-02-18 | 45 | -1027/+2015 |
| |\ \ | | | | | | | | | | | | | | | | | | | | | Conflicts: streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala streaming/src/main/scala/spark/streaming/dstream/NetworkInputDStream.scala | ||||
| | * | | Added checkpointing and fault-tolerance semantics to the programming guide. ↵ | Tathagata Das | 2013-02-18 | 6 | -6/+11 |
| | | | | | | | | | | | | | | | | Fixed default checkpoint interval to being a multiple of slide duration. Fixed visibility of some classes and objects to clean up docs. | ||||
| | * | | Many changes to ensure better 2nd recovery if 2nd failure happens while | Tathagata Das | 2013-02-17 | 18 | -97/+208 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | recovering from 1st failure - Made the scheduler to checkpoint after clearing old metadata which ensures that a new checkpoint is written as soon as at least one batch gets computed while recovering from a failure. This ensures that if there is a 2nd failure while recovering from 1st failure, the system start 2nd recovery from a newer checkpoint. - Modified Checkpoint writer to write checkpoint in a different thread. - Added a check to make sure that compute for InputDStreams gets called only for strictly increasing times. - Changed implementation of slice to call getOrCompute on parent DStream in time-increasing order. - Added testcase to test slice. - Fixed testGroupByKeyAndWindow testcase in JavaAPISuite to verify results with expected output in an order-independent manner. | ||||
| | * | | Made MasterFailureTest more robust. | Tathagata Das | 2013-02-15 | 1 | -4/+22 |
| | | | | |||||
| | * | | Moved Java streaming examples to examples/src/main/java/spark/streaming/... ↵ | Tathagata Das | 2013-02-14 | 1 | -1/+1 |
| | | | | | | | | | | | | | | | | and fixed logging in NetworkInputTracker to highlight errors when receiver deregisters/shuts down. |