diff options
| author | zsxwing <zsxwing@gmail.com> | 2015-01-11 16:23:28 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-01-11 16:23:56 -0800 |
| commit | 056149d10c64900fa15ed01843bf952ceb5708ea (patch) | |
| tree | d4c37f011c8165eb2c73c96ecc160cf1f0948322 /sql/hive-thriftserver | |
| parent | f04ff9d37ae312dccd16843df7d0e3a03c30e403 (diff) | |
| download | spark-056149d10c64900fa15ed01843bf952ceb5708ea.tar.gz spark-056149d10c64900fa15ed01843bf952ceb5708ea.tar.bz2 spark-056149d10c64900fa15ed01843bf952ceb5708ea.zip | |
[SPARK-4951][Core] Fix the issue that a busy executor may be killed
A few changes to fix this issue:
1. Handle the case that receiving `SparkListenerTaskStart` before `SparkListenerBlockManagerAdded`.
2. Don't add `executorId` to `removeTimes` when the executor is busy.
3. Use `HashMap.retain` to safely traverse the HashMap and remove items.
4. Use the same lock in ExecutorAllocationManager and ExecutorAllocationListener to fix the race condition in `totalPendingTasks`.
5. Move the blocking codes out of the message processing code in YarnSchedulerActor.
Author: zsxwing <zsxwing@gmail.com>
Closes #3783 from zsxwing/SPARK-4951 and squashes the following commits:
d51fa0d [zsxwing] Add comments
2e365ce [zsxwing] Remove expired executors from 'removeTimes' and add idle executors back when a new executor joins
49f61a9 [zsxwing] Eliminate duplicate executor registered warnings
d4c4e9a [zsxwing] Minor fixes for the code style
05f6238 [zsxwing] Move the blocking codes out of the message processing code
105ba3a [zsxwing] Fix the race condition in totalPendingTasks
d5c615d [zsxwing] Fix the issue that a busy executor may be killed
(cherry picked from commit 6942b974adad396cba2799eac1fa90448cea4da7)
Signed-off-by: Andrew Or <andrew@databricks.com>
Diffstat (limited to 'sql/hive-thriftserver')
0 files changed, 0 insertions, 0 deletions