aboutsummaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorTathagata Das <tathagata.das1565@gmail.com>2013-02-17 15:06:41 -0800
committerTathagata Das <tathagata.das1565@gmail.com>2013-02-17 15:06:41 -0800
commitf98c7da23ef66812b8b4888230ee98c07f09af23 (patch)
tree28aa7c6757dcdfe0ee72e95f93634edd77c89265 /docs
parentddcb976b0d7ce4a76168da33c0e947a5a6b5a255 (diff)
downloadspark-f98c7da23ef66812b8b4888230ee98c07f09af23.tar.gz
spark-f98c7da23ef66812b8b4888230ee98c07f09af23.tar.bz2
spark-f98c7da23ef66812b8b4888230ee98c07f09af23.zip
Many changes to ensure better 2nd recovery if 2nd failure happens while
recovering from 1st failure - Made the scheduler to checkpoint after clearing old metadata which ensures that a new checkpoint is written as soon as at least one batch gets computed while recovering from a failure. This ensures that if there is a 2nd failure while recovering from 1st failure, the system start 2nd recovery from a newer checkpoint. - Modified Checkpoint writer to write checkpoint in a different thread. - Added a check to make sure that compute for InputDStreams gets called only for strictly increasing times. - Changed implementation of slice to call getOrCompute on parent DStream in time-increasing order. - Added testcase to test slice. - Fixed testGroupByKeyAndWindow testcase in JavaAPISuite to verify results with expected output in an order-independent manner.
Diffstat (limited to 'docs')
0 files changed, 0 insertions, 0 deletions