diff options
| author | Shubham Chopra <schopra31@bloomberg.net> | 2017-03-28 09:47:29 +0800 |
|---|---|---|
| committer | Wenchen Fan <wenchen@databricks.com> | 2017-03-28 09:47:29 +0800 |
| commit | a250933c625ed720d15a0e479e9c51113605b102 (patch) | |
| tree | 00ed643ec67b61160c484f99dd1ea8907ad609df /.gitignore | |
| parent | 1d00761b9176a1f42976057ca78638c5b0763abc (diff) | |
| download | spark-a250933c625ed720d15a0e479e9c51113605b102.tar.gz spark-a250933c625ed720d15a0e479e9c51113605b102.tar.bz2 spark-a250933c625ed720d15a0e479e9c51113605b102.zip | |
[SPARK-19803][CORE][TEST] Proactive replication test failures
## What changes were proposed in this pull request?
Executors cache a list of their peers that is refreshed by default every minute. The cached stale references were randomly being used for replication. Since those executors were removed from the master, they did not occur in the block locations as reported by the master. This was fixed by
1. Refreshing peer cache in the block manager before trying to pro-actively replicate. This way the probability of replicating to a failed executor is eliminated.
2. Explicitly stopping the block manager in the tests. This shuts down the RPC endpoint use by the block manager. This way, even if a block manager tries to replicate using a stale reference, the replication logic should take care of refreshing the list of peers after failure.
## How was this patch tested?
Tested manually
Author: Shubham Chopra <schopra31@bloomberg.net>
Author: Kay Ousterhout <kayousterhout@gmail.com>
Author: Shubham Chopra <shubhamchopra@users.noreply.github.com>
Closes #17325 from shubhamchopra/SPARK-19803.
Diffstat (limited to '.gitignore')
0 files changed, 0 insertions, 0 deletions