diff options
| author | Tathagata Das <tathagata.das1565@gmail.com> | 2015-08-25 00:35:51 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-08-25 00:35:51 -0700 |
| commit | 1fc37581a52530bac5d555dbf14927a5780c3b75 (patch) | |
| tree | 03b493c92f0614cbf848dfade28e5d7ed82d4b07 /graphx | |
| parent | 57b960bf3706728513f9e089455a533f0244312e (diff) | |
| download | spark-1fc37581a52530bac5d555dbf14927a5780c3b75.tar.gz spark-1fc37581a52530bac5d555dbf14927a5780c3b75.tar.bz2 spark-1fc37581a52530bac5d555dbf14927a5780c3b75.zip | |
[SPARK-10210] [STREAMING] Filter out non-existent blocks before creating BlockRDD
When write ahead log is not enabled, a recovered streaming driver still tries to run jobs using pre-failure block ids, and fails as the block do not exists in-memory any more (and cannot be recovered as receiver WAL is not enabled).
This occurs because the driver-side WAL of ReceivedBlockTracker is recovers that past block information, and ReceiveInputDStream creates BlockRDDs even if those blocks do not exist.
The solution in this PR is to filter out block ids that do not exist before creating the BlockRDD. In addition, it adds unit tests to verify other logic in ReceiverInputDStream.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #8405 from tdas/SPARK-10210.
Diffstat (limited to 'graphx')
0 files changed, 0 insertions, 0 deletions