diff options
| author | Tathagata Das <tathagata.das1565@gmail.com> | 2014-10-30 14:51:13 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2014-10-30 14:51:13 -0700 |
| commit | 234de9232bcfa212317a8073c4a82c3863b36b14 (patch) | |
| tree | eb25ce0714c0b9151ccc25d7828f1383b5ba6477 /docs/ec2-scripts.md | |
| parent | d9327192eee7f18e92381c59a42b0e1770f1f8f4 (diff) | |
| download | spark-234de9232bcfa212317a8073c4a82c3863b36b14.tar.gz spark-234de9232bcfa212317a8073c4a82c3863b36b14.tar.bz2 spark-234de9232bcfa212317a8073c4a82c3863b36b14.zip | |
[SPARK-4028][Streaming] ReceivedBlockHandler interface to abstract the functionality of storage of received data
As part of the initiative to prevent data loss on streaming driver failure, this JIRA tracks the subtask of implementing a ReceivedBlockHandler, that abstracts the functionality of storage of received data blocks. The default implementation will maintain the current behavior of storing the data into BlockManager. The optional implementation will store the data to both BlockManager as well as a write ahead log.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #2940 from tdas/driver-ha-rbh and squashes the following commits:
78a4aaa [Tathagata Das] Fixed bug causing test failures.
f192f47 [Tathagata Das] Fixed import order.
df5f320 [Tathagata Das] Updated code to use ReceivedBlockStoreResult as the return type for handler's storeBlock
33c30c9 [Tathagata Das] Added license, and organized imports.
2f025b3 [Tathagata Das] Updates based on PR comments.
18aec1e [Tathagata Das] Moved ReceivedBlockInfo back into spark.streaming.scheduler package
95a4987 [Tathagata Das] Added ReceivedBlockHandler and its associated tests
Diffstat (limited to 'docs/ec2-scripts.md')
0 files changed, 0 insertions, 0 deletions