diff options
| author | Hemant Bhanawat <hemant@snappydata.io> | 2016-04-16 23:43:32 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-04-16 23:43:32 -0700 |
| commit | af1f4da76268115c5a4cc3035d3236ad27f7240a (patch) | |
| tree | af080d14f8f5f1172e67353d0350404f92506f99 /dev/.rat-excludes | |
| parent | 3394b12c379fe0a423d73dc6316aadca18cd2110 (diff) | |
| download | spark-af1f4da76268115c5a4cc3035d3236ad27f7240a.tar.gz spark-af1f4da76268115c5a4cc3035d3236ad27f7240a.tar.bz2 spark-af1f4da76268115c5a4cc3035d3236ad27f7240a.zip | |
[SPARK-13904][SCHEDULER] Add support for pluggable cluster manager
## What changes were proposed in this pull request?
This commit adds support for pluggable cluster manager. And also allows a cluster manager to clean up tasks without taking the parent process down.
To plug a new external cluster manager, ExternalClusterManager trait should be implemented. It returns task scheduler and backend scheduler that will be used by SparkContext to schedule tasks. An external cluster manager is registered using the java.util.ServiceLoader mechanism (This mechanism is also being used to register data sources like parquet, json, jdbc etc.). This allows auto-loading implementations of ExternalClusterManager interface.
Currently, when a driver fails, executors exit using system.exit. This does not bode well for cluster managers that would like to reuse the parent process of an executor. Hence,
1. Moving system.exit to a function that can be overriden in subclasses of CoarseGrainedExecutorBackend.
2. Added functionality of killing all the running tasks in an executor.
## How was this patch tested?
ExternalClusterManagerSuite.scala was added to test this patch.
Author: Hemant Bhanawat <hemant@snappydata.io>
Closes #11723 from hbhanawat/pluggableScheduler.
Diffstat (limited to 'dev/.rat-excludes')
| -rw-r--r-- | dev/.rat-excludes | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/dev/.rat-excludes b/dev/.rat-excludes index 8b5061415f..a409a3cb36 100644 --- a/dev/.rat-excludes +++ b/dev/.rat-excludes @@ -98,3 +98,4 @@ LZ4BlockInputStream.java spark-deps-.* .*csv .*tsv +org.apache.spark.scheduler.ExternalClusterManager |