diff options
| author | Andrew Or <andrewor14@gmail.com> | 2014-09-23 14:00:33 -0700 |
|---|---|---|
| committer | Andrew Or <andrewor14@gmail.com> | 2014-09-23 14:00:33 -0700 |
| commit | b3fef50e22fb3fe499f627179d17836a92dcb33a (patch) | |
| tree | a358cfc6d8d61062971720fbd7186e04e8bb1738 /core | |
| parent | d79238d03a2ffe0cf5fc6166543d67768693ddbe (diff) | |
| download | spark-b3fef50e22fb3fe499f627179d17836a92dcb33a.tar.gz spark-b3fef50e22fb3fe499f627179d17836a92dcb33a.tar.bz2 spark-b3fef50e22fb3fe499f627179d17836a92dcb33a.zip | |
[SPARK-3653] Respect SPARK_*_MEMORY for cluster mode
`SPARK_DRIVER_MEMORY` was only used to start the `SparkSubmit` JVM, which becomes the driver only in client mode but not cluster mode. In cluster mode, this property is simply not propagated to the worker nodes.
`SPARK_EXECUTOR_MEMORY` is picked up from `SparkContext`, but in cluster mode the driver runs on one of the worker machines, where this environment variable may not be set.
Author: Andrew Or <andrewor14@gmail.com>
Closes #2500 from andrewor14/memory-env-vars and squashes the following commits:
6217b38 [Andrew Or] Respect SPARK_*_MEMORY for cluster mode
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala index 92e0917743..2b72c61cc8 100644 --- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala +++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala @@ -75,6 +75,10 @@ private[spark] class SparkSubmitArguments(args: Seq[String]) { defaultProperties } + // Respect SPARK_*_MEMORY for cluster mode + driverMemory = sys.env.get("SPARK_DRIVER_MEMORY").orNull + executorMemory = sys.env.get("SPARK_EXECUTOR_MEMORY").orNull + parseOpts(args.toList) mergeSparkProperties() checkRequiredArguments() |