diff options
| author | Burak Yavuz <brkyvz@gmail.com> | 2017-04-12 11:24:59 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2017-04-12 11:24:59 -0700 |
| commit | 924c42477b5d6ed3c217c8eaaf4dc64b2379851a (patch) | |
| tree | 28209b8e1ad914b87812e07b08388c5ed16f6d51 /streaming | |
| parent | 99a9473127ec389283ac4ec3b721d2e34434e647 (diff) | |
| download | spark-924c42477b5d6ed3c217c8eaaf4dc64b2379851a.tar.gz spark-924c42477b5d6ed3c217c8eaaf4dc64b2379851a.tar.bz2 spark-924c42477b5d6ed3c217c8eaaf4dc64b2379851a.zip | |
[SPARK-20301][FLAKY-TEST] Fix Hadoop Shell.runCommand flakiness in Structured Streaming tests
## What changes were proposed in this pull request?
Some Structured Streaming tests show flakiness such as:
```
[info] - prune results by current_date, complete mode - 696 *** FAILED *** (10 seconds, 937 milliseconds)
[info] Timed out while stopping and waiting for microbatchthread to terminate.: The code passed to failAfter did not complete within 10 seconds.
```
This happens when we wait for the stream to stop, but it doesn't. The reason it doesn't stop is that we interrupt the microBatchThread, but Hadoop's `Shell.runCommand` swallows the interrupt exception, and the exception is not propagated upstream to the microBatchThread. Then this thread continues to run, only to start blocking on the `streamManualClock`.
## How was this patch tested?
Thousand retries locally and [Jenkins](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75720/testReport) of the flaky tests
Author: Burak Yavuz <brkyvz@gmail.com>
Closes #17613 from brkyvz/flaky-stream-agg.
Diffstat (limited to 'streaming')
0 files changed, 0 insertions, 0 deletions