diff options
author | Marcelo Vanzin <vanzin@cloudera.com> | 2016-09-06 15:54:54 -0700 |
---|---|---|
committer | Marcelo Vanzin <vanzin@cloudera.com> | 2016-09-06 15:54:54 -0700 |
commit | 0bd00ff2454c5046e4cb084ee64d432c4d3dcbc3 (patch) | |
tree | 9db958479bc72358c2082f000a914a2dce77f41b /yarn/src/test/scala/org | |
parent | 4f769b903bc9822c262f0a15f5933cc05c67923f (diff) | |
download | spark-0bd00ff2454c5046e4cb084ee64d432c4d3dcbc3.tar.gz spark-0bd00ff2454c5046e4cb084ee64d432c4d3dcbc3.tar.bz2 spark-0bd00ff2454c5046e4cb084ee64d432c4d3dcbc3.zip |
[SPARK-15891][YARN] Clean up some logging in the YARN AM.
To make the log file more readable, rework some of the logging done
by the AM:
- log executor command / env just once, since they're all almost the same;
the information that changes, such as executor ID, is already available
in other log messages.
- avoid printing logs when nothing happens, especially when updating the
container requests in the allocator.
- print fewer log messages when requesting many unlocalized executors,
instead of repeating the same message multiple times.
- removed some logs that seemed unnecessary.
In the process, I slightly fixed up the wording in a few log messages, and
did some minor clean up of method arguments that were redundant.
Tested by running existing unit tests, and analyzing the logs of an
application that exercises dynamic allocation by forcing executors
to be allocated and be killed in waves.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #14943 from vanzin/SPARK-15891.
Diffstat (limited to 'yarn/src/test/scala/org')
-rw-r--r-- | yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala b/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala index 207dbf56d3..696e552c35 100644 --- a/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala +++ b/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala @@ -19,12 +19,12 @@ package org.apache.spark.deploy.yarn import java.util.{Arrays, List => JList} -import org.apache.hadoop.conf.Configuration import org.apache.hadoop.fs.CommonConfigurationKeysPublic import org.apache.hadoop.net.DNSToSwitchMapping import org.apache.hadoop.yarn.api.records._ import org.apache.hadoop.yarn.client.api.AMRMClient import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest +import org.apache.hadoop.yarn.conf.YarnConfiguration import org.mockito.Mockito._ import org.scalatest.{BeforeAndAfterEach, Matchers} @@ -49,7 +49,7 @@ class MockResolver extends DNSToSwitchMapping { } class YarnAllocatorSuite extends SparkFunSuite with Matchers with BeforeAndAfterEach { - val conf = new Configuration() + val conf = new YarnConfiguration() conf.setClass( CommonConfigurationKeysPublic.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY, classOf[MockResolver], classOf[DNSToSwitchMapping]) |