diff options
| author | Patrick Wendell <pwendell@gmail.com> | 2014-01-12 20:04:21 -0800 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-01-12 20:04:21 -0800 |
| commit | 405bfe86ef9c3021358d2ac89192857478861fe0 (patch) | |
| tree | beb56b9652ca10c82d7bc165aadc43af54cda940 /NOTICE | |
| parent | 28a6b0cdbc75d58e36b1da3dcf257c61e44b0f7a (diff) | |
| parent | aa2c993858f87adc249eb9c20a908a125f8f4033 (diff) | |
| download | spark-405bfe86ef9c3021358d2ac89192857478861fe0.tar.gz spark-405bfe86ef9c3021358d2ac89192857478861fe0.tar.bz2 spark-405bfe86ef9c3021358d2ac89192857478861fe0.zip | |
Merge pull request #394 from tdas/error-handling
Better error handling in Spark Streaming and more API cleanup
Earlier errors in jobs generated by Spark Streaming (or in the generation of jobs) could not be caught from the main driver thread (i.e. the thread that called StreamingContext.start()) as it would be thrown in different threads. With this change, after `ssc.start`, one can call `ssc.awaitTermination()` which will be block until the ssc is closed, or there is an exception. This makes it easier to debug.
This change also adds ssc.stop(<stop-spark-context>) where you can stop StreamingContext without stopping the SparkContext.
Also fixes the bug that came up with PRs #393 and #381. MetadataCleaner default value has been changed from 3500 to -1 for normal SparkContext and 3600 when creating a StreamingContext. Also, updated StreamingListenerBus with changes similar to SparkListenerBus in #392.
And changed a lot of protected[streaming] to private[streaming].
Diffstat (limited to 'NOTICE')
0 files changed, 0 insertions, 0 deletions