diff options
| author | jinxing <jinxing6042@126.com> | 2017-03-23 23:25:56 -0700 |
|---|---|---|
| committer | Kay Ousterhout <kayousterhout@gmail.com> | 2017-03-23 23:25:56 -0700 |
| commit | 19596c28b6ef6e7abe0cfccfd2269c2fddf1fdee (patch) | |
| tree | 3634264ca51c7cb76ae1b6c8de710fdeed08ac0a /streaming/src | |
| parent | bb823ca4b479a00030c4919c2d857d254b2a44d8 (diff) | |
| download | spark-19596c28b6ef6e7abe0cfccfd2269c2fddf1fdee.tar.gz spark-19596c28b6ef6e7abe0cfccfd2269c2fddf1fdee.tar.bz2 spark-19596c28b6ef6e7abe0cfccfd2269c2fddf1fdee.zip | |
[SPARK-16929] Improve performance when check speculatable tasks.
## What changes were proposed in this pull request?
1. Use a MedianHeap to record durations of successful tasks. When check speculatable tasks, we can get the median duration with O(1) time complexity.
2. `checkSpeculatableTasks` will synchronize `TaskSchedulerImpl`. If `checkSpeculatableTasks` doesn't finish with 100ms, then the possibility exists for that thread to release and then immediately re-acquire the lock. Change `scheduleAtFixedRate` to be `scheduleWithFixedDelay` when call method of `checkSpeculatableTasks`.
## How was this patch tested?
Added MedianHeapSuite.
Author: jinxing <jinxing6042@126.com>
Closes #16867 from jinxing64/SPARK-16929.
Diffstat (limited to 'streaming/src')
0 files changed, 0 insertions, 0 deletions