diff options
| author | Tathagata Das <tathagata.das1565@gmail.com> | 2015-04-23 11:29:34 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-04-23 11:29:34 -0700 |
| commit | 534f2a43625fbf1a3a65d09550a19875cd1dce43 (patch) | |
| tree | 8f77495057454cd0ee360e00841f6df84afcb3b5 /sql | |
| parent | cc48e6387abdd909921cb58e0588cdf226556bcd (diff) | |
| download | spark-534f2a43625fbf1a3a65d09550a19875cd1dce43.tar.gz spark-534f2a43625fbf1a3a65d09550a19875cd1dce43.tar.bz2 spark-534f2a43625fbf1a3a65d09550a19875cd1dce43.zip | |
[SPARK-6752][Streaming] Allow StreamingContext to be recreated from checkpoint and existing SparkContext
Currently if you want to create a StreamingContext from checkpoint information, the system will create a new SparkContext. This prevent StreamingContext to be recreated from checkpoints in managed environments where SparkContext is precreated.
The solution in this PR: Introduce the following methods on StreamingContext
1. `new StreamingContext(checkpointDirectory, sparkContext)`
Recreate StreamingContext from checkpoint using the provided SparkContext
2. `StreamingContext.getOrCreate(checkpointDirectory, sparkContext, createFunction: SparkContext => StreamingContext)`
If checkpoint file exists, then recreate StreamingContext using the provided SparkContext (that is, call 1.), else create StreamingContext using the provided createFunction
TODO: the corresponding Java and Python API has to be added as well.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #5428 from tdas/SPARK-6752 and squashes the following commits:
94db63c [Tathagata Das] Fix long line.
524f519 [Tathagata Das] Many changes based on PR comments.
eabd092 [Tathagata Das] Added Function0, Java API and unit tests for StreamingContext.getOrCreate
36a7823 [Tathagata Das] Minor changes.
204814e [Tathagata Das] Added StreamingContext.getOrCreate with existing SparkContext
Diffstat (limited to 'sql')
0 files changed, 0 insertions, 0 deletions