diff options
| author | Tathagata Das <tathagata.das1565@gmail.com> | 2015-05-05 01:45:19 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-05-05 01:45:19 -0700 |
| commit | 1854ac326a9cc6014817d8df30ed0458eee5d7d1 (patch) | |
| tree | b9fc9a06d93207ce3f99ec73d6f10367e526e5fe /tools | |
| parent | c5790a2f772168351c18bb0da51a124cee89a06f (diff) | |
| download | spark-1854ac326a9cc6014817d8df30ed0458eee5d7d1.tar.gz spark-1854ac326a9cc6014817d8df30ed0458eee5d7d1.tar.bz2 spark-1854ac326a9cc6014817d8df30ed0458eee5d7d1.zip | |
[SPARK-7139] [STREAMING] Allow received block metadata to be saved to WAL and recovered on driver failure
- Enabled ReceivedBlockTracker WAL by default
- Stored block metadata in the WAL
- Optimized WALBackedBlockRDD by skipping block fetch when the block is known to not exist in Spark
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #5732 from tdas/SPARK-7139 and squashes the following commits:
575476e [Tathagata Das] Added more tests to get 100% coverage of the WALBackedBlockRDD
19668ba [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-7139
685fab3 [Tathagata Das] Addressed comments in PR
637bc9c [Tathagata Das] Changed segment to handle
466212c [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-7139
5f67a59 [Tathagata Das] Fixed HdfsUtils to handle append in local file system
1bc5bc3 [Tathagata Das] Fixed bug on unexpected recovery
d06fa21 [Tathagata Das] Enabled ReceivedBlockTracker by default, stored block metadata and optimized block fetching in WALBackedBlockRDD
Diffstat (limited to 'tools')
0 files changed, 0 insertions, 0 deletions