diff options
| author | Sandy Ryza <sandy@cloudera.com> | 2014-04-07 13:28:14 -0500 |
|---|---|---|
| committer | Thomas Graves <tgraves@apache.org> | 2014-04-07 13:28:14 -0500 |
| commit | 9dd8b9166225cbcfcab1f39268244c2feb42a658 (patch) | |
| tree | 85210a0b181ec29febaf95f01d1717b321abbbe8 /yarn/stable | |
| parent | 83f2a2f14e4145a04672e42216d43100a66b1fc2 (diff) | |
| download | spark-9dd8b9166225cbcfcab1f39268244c2feb42a658.tar.gz spark-9dd8b9166225cbcfcab1f39268244c2feb42a658.tar.bz2 spark-9dd8b9166225cbcfcab1f39268244c2feb42a658.zip | |
SPARK-1252. On YARN, use container-log4j.properties for executors
container-log4j.properties is a file that YARN provides so that containers can have log4j.properties distinct from that of the NodeManagers.
Logs now go to syslog, and stderr and stdout just have the process's standard err and standard out.
I tested this on pseudo-distributed clusters for both yarn (Hadoop 2.2) and yarn-alpha (Hadoop 0.23.7)/
Author: Sandy Ryza <sandy@cloudera.com>
Closes #148 from sryza/sandy-spark-1252 and squashes the following commits:
c0043b8 [Sandy Ryza] Put log4j.properties file under common
55823da [Sandy Ryza] Add license headers to new files
10934b8 [Sandy Ryza] Add log4j-spark-container.properties and support SPARK_LOG4J_CONF
e74450b [Sandy Ryza] SPARK-1252. On YARN, use container-log4j.properties for executors
Diffstat (limited to 'yarn/stable')
| -rw-r--r-- | yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala b/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala index 53c403f7d0..81d9d1b5c9 100644 --- a/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala +++ b/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala @@ -78,7 +78,8 @@ class ExecutorRunnable( credentials.writeTokenStorageToStream(dob) ctx.setTokens(ByteBuffer.wrap(dob.getData())) - val commands = prepareCommand(masterAddress, slaveId, hostname, executorMemory, executorCores) + val commands = prepareCommand(masterAddress, slaveId, hostname, executorMemory, executorCores, + localResources.contains(ClientBase.LOG4J_PROP)) logInfo("Setting up executor with commands: " + commands) ctx.setCommands(commands) |