diff options
author | Andrew Or <andrew@databricks.com> | 2014-12-10 14:27:53 -0800 |
---|---|---|
committer | Andrew Or <andrew@databricks.com> | 2014-12-10 14:27:53 -0800 |
commit | 4f93d0cabe5d1fc7c0fd0a33d992fd85df1fecb4 (patch) | |
tree | b5886d48a7bcacdd8b7b6423547367277829320a /external | |
parent | 447ae2de5d4c2af865fdb63f8b876b865de60f74 (diff) | |
download | spark-4f93d0cabe5d1fc7c0fd0a33d992fd85df1fecb4.tar.gz spark-4f93d0cabe5d1fc7c0fd0a33d992fd85df1fecb4.tar.bz2 spark-4f93d0cabe5d1fc7c0fd0a33d992fd85df1fecb4.zip |
[SPARK-4759] Fix driver hanging from coalescing partitions
The driver hangs sometimes when we coalesce RDD partitions. See JIRA for more details and reproduction.
This is because our use of empty string as default preferred location in `CoalescedRDDPartition` causes the `TaskSetManager` to schedule the corresponding task on host `""` (empty string). The intended semantics here, however, is that the partition does not have a preferred location, and the TSM should schedule the corresponding task accordingly.
Author: Andrew Or <andrew@databricks.com>
Closes #3633 from andrewor14/coalesce-preferred-loc and squashes the following commits:
e520d6b [Andrew Or] Oops
3ebf8bd [Andrew Or] A few comments
f370a4e [Andrew Or] Fix tests
2f7dfb6 [Andrew Or] Avoid using empty string as default preferred location
Diffstat (limited to 'external')
0 files changed, 0 insertions, 0 deletions