diff options
author | Aaron Davidson <aaron@databricks.com> | 2014-11-06 18:39:14 -0800 |
---|---|---|
committer | Reynold Xin <rxin@databricks.com> | 2014-11-06 18:39:14 -0800 |
commit | f165b2bbf5d4acf34d826fa55b900f5bbc295654 (patch) | |
tree | 083430602410d4d9c1884b2092aa599a24cc35a4 /pom.xml | |
parent | 6e9ef10fd7446a11f37446c961916ba2a8e02cb8 (diff) | |
download | spark-f165b2bbf5d4acf34d826fa55b900f5bbc295654.tar.gz spark-f165b2bbf5d4acf34d826fa55b900f5bbc295654.tar.bz2 spark-f165b2bbf5d4acf34d826fa55b900f5bbc295654.zip |
[SPARK-4188] [Core] Perform network-level retry of shuffle file fetches
This adds a RetryingBlockFetcher to the NettyBlockTransferService which is wrapped around our typical OneForOneBlockFetcher, adding retry logic in the event of an IOException.
This sort of retry allows us to avoid marking an entire executor as failed due to garbage collection or high network load.
TODO:
- [x] unit tests
- [x] put in ExternalShuffleClient too
Author: Aaron Davidson <aaron@databricks.com>
Closes #3101 from aarondav/retry and squashes the following commits:
72a2a32 [Aaron Davidson] Add that we should remove the condition around the retry thingy
c7fd107 [Aaron Davidson] Fix unit tests
e80e4c2 [Aaron Davidson] Address initial comments
6f594cd [Aaron Davidson] Fix unit test
05ff43c [Aaron Davidson] Add to external shuffle client and add unit test
66e5a24 [Aaron Davidson] [SPARK-4238] [Core] Perform network-level retry of shuffle file fetches
Diffstat (limited to 'pom.xml')
0 files changed, 0 insertions, 0 deletions