diff options
| author | vundela <vsr@cloudera.com> | 2015-10-20 11:12:28 -0700 |
|---|---|---|
| committer | Marcelo Vanzin <vanzin@cloudera.com> | 2015-10-20 11:12:28 -0700 |
| commit | 2f6dd634c169a34811738a3c230cce4462612748 (patch) | |
| tree | 32749f98c4df0f5970ba7d0cb2e3e3b11684593d /yarn | |
| parent | e18b571c3374ecbfc0b20a5064cb58d57a2a7d21 (diff) | |
| download | spark-2f6dd634c169a34811738a3c230cce4462612748.tar.gz spark-2f6dd634c169a34811738a3c230cce4462612748.tar.bz2 spark-2f6dd634c169a34811738a3c230cce4462612748.zip | |
[SPARK-11105] [YARN] Distribute log4j.properties to executors
Currently log4j.properties file is not uploaded to executor's which is leading them to use the default values. This fix will make sure that file is always uploaded to distributed cache so that executor will use the latest settings.
If user specifies log configurations through --files then executors will be picking configs from --files instead of $SPARK_CONF_DIR/log4j.properties
Author: vundela <vsr@cloudera.com>
Author: Srinivasa Reddy Vundela <vsr@cloudera.com>
Closes #9118 from vundela/master.
Diffstat (limited to 'yarn')
| -rw-r--r-- | yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 13 |
1 files changed, 13 insertions, 0 deletions
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala index 754215db2a..4954b61809 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala @@ -497,6 +497,19 @@ private[spark] class Client( */ private def createConfArchive(): File = { val hadoopConfFiles = new HashMap[String, File]() + + // Uploading $SPARK_CONF_DIR/log4j.properties file to the distributed cache to make sure that + // the executors will use the latest configurations instead of the default values. This is + // required when user changes log4j.properties directly to set the log configurations. If + // configuration file is provided through --files then executors will be taking configurations + // from --files instead of $SPARK_CONF_DIR/log4j.properties. + val log4jFileName = "log4j.properties" + Option(Utils.getContextOrSparkClassLoader.getResource(log4jFileName)).foreach { url => + if (url.getProtocol == "file") { + hadoopConfFiles(log4jFileName) = new File(url.getPath) + } + } + Seq("HADOOP_CONF_DIR", "YARN_CONF_DIR").foreach { envKey => sys.env.get(envKey).foreach { path => val dir = new File(path) |