diff options
| author | Matei Zaharia <matei@databricks.com> | 2013-12-24 00:08:48 -0500 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2013-12-24 00:08:48 -0500 |
| commit | 23a9ae6be33ab2721005b766615660e07eec739f (patch) | |
| tree | fbc3564d1949c8517aebe613ebc09c8e79a8e627 /spark-class | |
| parent | 11107c9de524a7397ec99333f5d7b00886217781 (diff) | |
| parent | 6eaa0505493511adb040257abc749fcd774bbb68 (diff) | |
| download | spark-23a9ae6be33ab2721005b766615660e07eec739f.tar.gz spark-23a9ae6be33ab2721005b766615660e07eec739f.tar.bz2 spark-23a9ae6be33ab2721005b766615660e07eec739f.zip | |
Merge pull request #277 from tdas/scheduler-update
Refactored the streaming scheduler and added StreamingListener interface
- Refactored the streaming scheduler for cleaner code. Specifically, the JobManager was renamed to JobScheduler, as it does the actual scheduling of Spark jobs to the SparkContext. The earlier Scheduler was renamed to JobGenerator, as it actually generates the jobs from the DStreams. The JobScheduler starts the JobGenerator. Also, moved all the scheduler related code from spark.streaming to spark.streaming.scheduler package.
- Implemented the StreamingListener interface, similar to SparkListener. The streaming version of StatusReportListener prints the batch processing time statistics (for now). Added StreamingListernerSuite to test it.
- Refactored streaming TestSuiteBase for deduping code in the other streaming testsuites.
Diffstat (limited to 'spark-class')
0 files changed, 0 insertions, 0 deletions