diff options
| author | CodingCat <zhunansjtu@gmail.com> | 2014-08-05 23:02:58 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-08-05 23:02:58 -0700 |
| commit | 63bdb1f41b4895e3a9444f7938094438a94d3007 (patch) | |
| tree | 75aaa960879e75c0d51f55deffdefbe1217c41ce /external | |
| parent | 5a826c00c3255a2d9e5eb17d6b1abf83f7c1a08d (diff) | |
| download | spark-63bdb1f41b4895e3a9444f7938094438a94d3007.tar.gz spark-63bdb1f41b4895e3a9444f7938094438a94d3007.tar.bz2 spark-63bdb1f41b4895e3a9444f7938094438a94d3007.zip | |
SPARK-2294: fix locality inversion bug in TaskManager
copied from original JIRA (https://issues.apache.org/jira/browse/SPARK-2294):
If an executor E is free, a task may be speculatively assigned to E when there are other tasks in the job that have not been launched (at all) yet. Similarly, a task without any locality preferences may be assigned to E when there was another NODE_LOCAL task that could have been scheduled.
This happens because TaskSchedulerImpl calls TaskSetManager.resourceOffer (which in turn calls TaskSetManager.findTask) with increasing locality levels, beginning with PROCESS_LOCAL, followed by NODE_LOCAL, and so on until the highest currently allowed level. Now, supposed NODE_LOCAL is the highest currently allowed locality level. The first time findTask is called, it will be called with max level PROCESS_LOCAL; if it cannot find any PROCESS_LOCAL tasks, it will try to schedule tasks with no locality preferences or speculative tasks. As a result, speculative tasks or tasks with no preferences may be scheduled instead of NODE_LOCAL tasks.
----
I added an additional parameter in resourceOffer and findTask, maxLocality, indicating when we should consider the tasks without locality preference
Author: CodingCat <zhunansjtu@gmail.com>
Closes #1313 from CodingCat/SPARK-2294 and squashes the following commits:
bf3f13b [CodingCat] rollback some forgotten changes
89f9bc0 [CodingCat] address matei's comments
18cae02 [CodingCat] add test case for node-local tasks
2ba6195 [CodingCat] fix failed test cases
87dd09e [CodingCat] fix style
9b9432f [CodingCat] remove hasNodeLocalOnlyTasks
fdd1573 [CodingCat] fix failed test cases
941a4fd [CodingCat] see my shocked face..........
f600085 [CodingCat] remove hasNodeLocalOnlyTasks checking
0b8a46b [CodingCat] test whether hasNodeLocalOnlyTasks affect the results
73ceda8 [CodingCat] style fix
b3a430b [CodingCat] remove fine granularity tracking for node-local only tasks
f9a2ad8 [CodingCat] simplify the logic in TaskSchedulerImpl
c8c1de4 [CodingCat] simplify the patch
be652ed [CodingCat] avoid unnecessary delay when we only have nopref tasks
dee9e22 [CodingCat] fix locality inversion bug in TaskManager by moving nopref branch
Diffstat (limited to 'external')
0 files changed, 0 insertions, 0 deletions