diff options
author | Timothy Chen <tnachen@gmail.com> | 2015-08-03 01:55:58 -0700 |
---|---|---|
committer | Andrew Or <andrew@databricks.com> | 2015-08-03 01:55:58 -0700 |
commit | 95dccc63350c45045f038bab9f8a5080b4e1f8cc (patch) | |
tree | c6ac9b266234a3d349ce4ada3a80a17adc7744fc /launcher/src/main/java | |
parent | 1ebd41b141a95ec264bd2dd50f0fe24cd459035d (diff) | |
download | spark-95dccc63350c45045f038bab9f8a5080b4e1f8cc.tar.gz spark-95dccc63350c45045f038bab9f8a5080b4e1f8cc.tar.bz2 spark-95dccc63350c45045f038bab9f8a5080b4e1f8cc.zip |
[SPARK-8873] [MESOS] Clean up shuffle files if external shuffle service is used
This patch builds directly on #7820, which is largely written by tnachen. The only addition is one commit for cleaning up the code. There should be no functional differences between this and #7820.
Author: Timothy Chen <tnachen@gmail.com>
Author: Andrew Or <andrew@databricks.com>
Closes #7881 from andrewor14/tim-cleanup-mesos-shuffle and squashes the following commits:
8894f7d [Andrew Or] Clean up code
2a5fa10 [Andrew Or] Merge branch 'mesos_shuffle_clean' of github.com:tnachen/spark into tim-cleanup-mesos-shuffle
fadff89 [Timothy Chen] Address comments.
e4d0f1d [Timothy Chen] Clean up external shuffle data on driver exit with Mesos.
Diffstat (limited to 'launcher/src/main/java')
-rw-r--r-- | launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java b/launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java index de85720feb..5f95e2c74f 100644 --- a/launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java +++ b/launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java @@ -69,7 +69,8 @@ class SparkClassCommandBuilder extends AbstractCommandBuilder { } else if (className.equals("org.apache.spark.executor.MesosExecutorBackend")) { javaOptsKeys.add("SPARK_EXECUTOR_OPTS"); memKey = "SPARK_EXECUTOR_MEMORY"; - } else if (className.equals("org.apache.spark.deploy.ExternalShuffleService")) { + } else if (className.equals("org.apache.spark.deploy.ExternalShuffleService") || + className.equals("org.apache.spark.deploy.mesos.MesosExternalShuffleService")) { javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS"); javaOptsKeys.add("SPARK_SHUFFLE_OPTS"); memKey = "SPARK_DAEMON_MEMORY"; |