diff options
author | Tathagata Das <tathagata.das1565@gmail.com> | 2013-02-17 15:06:41 -0800 |
---|---|---|
committer | Tathagata Das <tathagata.das1565@gmail.com> | 2013-02-17 15:06:41 -0800 |
commit | f98c7da23ef66812b8b4888230ee98c07f09af23 (patch) | |
tree | 28aa7c6757dcdfe0ee72e95f93634edd77c89265 /docs/python-programming-guide.md | |
parent | ddcb976b0d7ce4a76168da33c0e947a5a6b5a255 (diff) | |
download | spark-f98c7da23ef66812b8b4888230ee98c07f09af23.tar.gz spark-f98c7da23ef66812b8b4888230ee98c07f09af23.tar.bz2 spark-f98c7da23ef66812b8b4888230ee98c07f09af23.zip |
Many changes to ensure better 2nd recovery if 2nd failure happens while
recovering from 1st failure
- Made the scheduler to checkpoint after clearing old metadata which
ensures that a new checkpoint is written as soon as at least one batch
gets computed while recovering from a failure. This ensures that if
there is a 2nd failure while recovering from 1st failure, the system
start 2nd recovery from a newer checkpoint.
- Modified Checkpoint writer to write checkpoint in a different thread.
- Added a check to make sure that compute for InputDStreams gets called
only for strictly increasing times.
- Changed implementation of slice to call getOrCompute on parent DStream
in time-increasing order.
- Added testcase to test slice.
- Fixed testGroupByKeyAndWindow testcase in JavaAPISuite to verify
results with expected output in an order-independent manner.
Diffstat (limited to 'docs/python-programming-guide.md')
0 files changed, 0 insertions, 0 deletions