--- layout: global title: Running Spark on YARN --- Support for running on [YARN (Hadoop NextGen)](http://hadoop.apache.org/docs/r2.0.2-alpha/hadoop-yarn/hadoop-yarn-site/YARN.html) was added to Spark in version 0.6.0, and improved in subsequent releases. # Preparations Running Spark-on-YARN requires a binary distribution of Spark which is built with YARN support. Binary distributions can be downloaded from the Spark project website. To build Spark yourself, refer to the [building with Maven guide](building-with-maven.html). # Configuration Most of the configs are the same for Spark on YARN as for other deployment modes. See the [configuration page](configuration.html) for more information on those. These are configs that are specific to Spark on YARN. #### Environment Variables * `SPARK_YARN_USER_ENV`, to add environment variables to the Spark processes launched on YARN. This can be a comma separated list of environment variables, e.g. `SPARK_YARN_USER_ENV="JAVA_HOME=/jdk64,FOO=bar"`. #### Spark Properties
Property Name | Default | Meaning |
---|---|---|
spark.yarn.applicationMaster.waitTries |
10 | Set the number of times the ApplicationMaster waits for the the Spark master and then also the number of tries it waits for the SparkContext to be initialized |
spark.yarn.submit.file.replication |
3 | HDFS replication level for the files uploaded into HDFS for the application. These include things like the Spark jar, the app jar, and any distributed cache files/archives. |
spark.yarn.preserve.staging.files |
false | Set to true to preserve the staged files (Spark jar, app jar, distributed cache files) at the end of the job rather then delete them. |
spark.yarn.scheduler.heartbeat.interval-ms |
5000 | The interval in ms in which the Spark application master heartbeats into the YARN ResourceManager. |
spark.yarn.max.executor.failures |
2*numExecutors | The maximum number of executor failures before failing the application. |
spark.yarn.historyServer.address |
(none) | The address of the Spark history server (i.e. host.com:18080). The address should not contain a scheme (http://). Defaults to not being set since the history server is an optional service. This address is given to the YARN ResourceManager when the Spark application finishes to link the application from the ResourceManager UI to the Spark history server UI. |