From af1f4da76268115c5a4cc3035d3236ad27f7240a Mon Sep 17 00:00:00 2001 From: Hemant Bhanawat Date: Sat, 16 Apr 2016 23:43:32 -0700 Subject: [SPARK-13904][SCHEDULER] Add support for pluggable cluster manager ## What changes were proposed in this pull request? This commit adds support for pluggable cluster manager. And also allows a cluster manager to clean up tasks without taking the parent process down. To plug a new external cluster manager, ExternalClusterManager trait should be implemented. It returns task scheduler and backend scheduler that will be used by SparkContext to schedule tasks. An external cluster manager is registered using the java.util.ServiceLoader mechanism (This mechanism is also being used to register data sources like parquet, json, jdbc etc.). This allows auto-loading implementations of ExternalClusterManager interface. Currently, when a driver fails, executors exit using system.exit. This does not bode well for cluster managers that would like to reuse the parent process of an executor. Hence, 1. Moving system.exit to a function that can be overriden in subclasses of CoarseGrainedExecutorBackend. 2. Added functionality of killing all the running tasks in an executor. ## How was this patch tested? ExternalClusterManagerSuite.scala was added to test this patch. Author: Hemant Bhanawat Closes #11723 from hbhanawat/pluggableScheduler. --- dev/.rat-excludes | 1 + 1 file changed, 1 insertion(+) (limited to 'dev') diff --git a/dev/.rat-excludes b/dev/.rat-excludes index 8b5061415f..a409a3cb36 100644 --- a/dev/.rat-excludes +++ b/dev/.rat-excludes @@ -98,3 +98,4 @@ LZ4BlockInputStream.java spark-deps-.* .*csv .*tsv +org.apache.spark.scheduler.ExternalClusterManager -- cgit v1.2.3