diff options
author | Imran Rashid <irashid@cloudera.com> | 2016-10-12 16:43:03 -0500 |
---|---|---|
committer | Imran Rashid <irashid@cloudera.com> | 2016-10-12 16:43:03 -0500 |
commit | 9ce7d3e542e786c62f047c13f3001e178f76e06a (patch) | |
tree | 6d43e48a1d969fb70347b8540b0bb50e4456b6d6 /python/pyspark/sql/window.py | |
parent | 47776e7c0c68590fe446cef910900b1aaead06f9 (diff) | |
download | spark-9ce7d3e542e786c62f047c13f3001e178f76e06a.tar.gz spark-9ce7d3e542e786c62f047c13f3001e178f76e06a.tar.bz2 spark-9ce7d3e542e786c62f047c13f3001e178f76e06a.zip |
[SPARK-17675][CORE] Expand Blacklist for TaskSets
## What changes were proposed in this pull request?
This is a step along the way to SPARK-8425.
To enable incremental review, the first step proposed here is to expand the blacklisting within tasksets. In particular, this will enable blacklisting for
* (task, executor) pairs (this already exists via an undocumented config)
* (task, node)
* (taskset, executor)
* (taskset, node)
Adding (task, node) is critical to making spark fault-tolerant of one-bad disk in a cluster, without requiring careful tuning of "spark.task.maxFailures". The other additions are also important to avoid many misleading task failures and long scheduling delays when there is one bad node on a large cluster.
Note that some of the code changes here aren't really required for just this -- they put pieces in place for SPARK-8425 even though they are not used yet (eg. the `BlacklistTracker` helper is a little out of place, `TaskSetBlacklist` holds onto a little more info than it needs to for just this change, and `ExecutorFailuresInTaskSet` is more complex than it needs to be).
## How was this patch tested?
Added unit tests, run tests via jenkins.
Author: Imran Rashid <irashid@cloudera.com>
Author: mwws <wei.mao@intel.com>
Closes #15249 from squito/taskset_blacklist_only.
Diffstat (limited to 'python/pyspark/sql/window.py')
0 files changed, 0 insertions, 0 deletions