diff options
author | w00228970 <wangfei1@huawei.com> | 2016-09-28 12:02:59 -0700 |
---|---|---|
committer | Shixiong Zhu <shixiong@databricks.com> | 2016-09-28 12:02:59 -0700 |
commit | 46d1203bf2d01b219c4efc7e0e77a844c0c664da (patch) | |
tree | 37c219e3d7f92dde99870543b694b3dbc77144ec /extras/kinesis-asl-assembly | |
parent | 2190037757a81d3172f75227f7891d968e1f0d90 (diff) | |
download | spark-46d1203bf2d01b219c4efc7e0e77a844c0c664da.tar.gz spark-46d1203bf2d01b219c4efc7e0e77a844c0c664da.tar.bz2 spark-46d1203bf2d01b219c4efc7e0e77a844c0c664da.zip |
[SPARK-17644][CORE] Do not add failedStages when abortStage for fetch failure
## What changes were proposed in this pull request?
| Time |Thread 1 , Job1 | Thread 2 , Job2 |
|:-------------:|:-------------:|:-----:|
| 1 | abort stage due to FetchFailed | |
| 2 | failedStages += failedStage | |
| 3 | | task failed due to FetchFailed |
| 4 | | can not post ResubmitFailedStages because failedStages is not empty |
Then job2 of thread2 never resubmit the failed stage and hang.
We should not add the failedStages when abortStage for fetch failure
## How was this patch tested?
added unit test
Author: w00228970 <wangfei1@huawei.com>
Author: wangfei <wangfei_hello@126.com>
Closes #15213 from scwf/dag-resubmit.
Diffstat (limited to 'extras/kinesis-asl-assembly')
0 files changed, 0 insertions, 0 deletions