diff options
| author | Ryan Blue <blue@apache.org> | 2016-03-14 11:18:32 -0700 |
|---|---|---|
| committer | Marcelo Vanzin <vanzin@cloudera.com> | 2016-03-14 11:18:37 -0700 |
| commit | 63f642aea31fe0d202ce585681d51e7ac1715ba7 (patch) | |
| tree | 92ec9f8cdcc3cca1db3aa6dab9f0764d2a8081b0 /R/.gitignore | |
| parent | 45f8053be5c635b50c7b4ef5a0dc75d30f411291 (diff) | |
| download | spark-63f642aea31fe0d202ce585681d51e7ac1715ba7.tar.gz spark-63f642aea31fe0d202ce585681d51e7ac1715ba7.tar.bz2 spark-63f642aea31fe0d202ce585681d51e7ac1715ba7.zip | |
[SPARK-13779][YARN] Avoid cancelling non-local container requests.
To maximize locality, the YarnAllocator would cancel any requests with a
stale locality preference or no locality preference. This assumed that
the majority of tasks had locality preferences, but may not be the case
when scanning S3. This caused container requests for S3 tasks to be
constantly cancelled and resubmitted.
This changes the allocator's logic to cancel only stale requests and
just enough requests without locality preferences to submit requests
with locality preferences. This avoids cancelling requests without
locality preferences that would be resubmitted without locality
preferences.
We've deployed this patch on our clusters and verified that jobs that couldn't get executors because they kept canceling and resubmitting requests are fixed. Large jobs are running fine.
Author: Ryan Blue <blue@apache.org>
Closes #11612 from rdblue/SPARK-13779-fix-yarn-allocator-requests.
Diffstat (limited to 'R/.gitignore')
0 files changed, 0 insertions, 0 deletions