diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2014-08-18 14:10:10 -0700 |
|---|---|---|
| committer | Andrew Or <andrewor14@gmail.com> | 2014-08-18 14:10:10 -0700 |
| commit | 6201b27643023569e19b68aa9d5c4e4e59ce0d79 (patch) | |
| tree | 3404b653009270202c8dc1409bd94d6ece2b9600 /core | |
| parent | d1d0ee41c27f1d07fed0c5d56ba26c723cc3dc26 (diff) | |
| download | spark-6201b27643023569e19b68aa9d5c4e4e59ce0d79.tar.gz spark-6201b27643023569e19b68aa9d5c4e4e59ce0d79.tar.bz2 spark-6201b27643023569e19b68aa9d5c4e4e59ce0d79.zip | |
[SPARK-2718] [yarn] Handle quotes and other characters in user args.
Due to the way Yarn runs things through bash, normal quoting doesn't
work as expected. This change applies the necessary voodoo to the user
args to avoid issues with bash and special characters.
The change also uncovered an issue with the event logger app name
sanitizing code; it wasn't cleaning up all "bad" characters, so
sometimes it would fail to create the log dirs. I just added some
more bad character replacements.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #1724 from vanzin/SPARK-2718 and squashes the following commits:
cc84b89 [Marcelo Vanzin] Review feedback.
c1a257a [Marcelo Vanzin] Add test for backslashes.
55571d4 [Marcelo Vanzin] Unbreak yarn-client.
515613d [Marcelo Vanzin] [SPARK-2718] [yarn] Handle quotes and other characters in user args.
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala b/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala index 7378ce923f..370fcd85aa 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala @@ -54,7 +54,8 @@ private[spark] class EventLoggingListener( private val testing = sparkConf.getBoolean("spark.eventLog.testing", false) private val outputBufferSize = sparkConf.getInt("spark.eventLog.buffer.kb", 100) * 1024 private val logBaseDir = sparkConf.get("spark.eventLog.dir", DEFAULT_LOG_DIR).stripSuffix("/") - private val name = appName.replaceAll("[ :/]", "-").toLowerCase + "-" + System.currentTimeMillis + private val name = appName.replaceAll("[ :/]", "-").replaceAll("[${}'\"]", "_") + .toLowerCase + "-" + System.currentTimeMillis val logDir = Utils.resolveURI(logBaseDir) + "/" + name.stripSuffix("/") protected val logger = new FileLogger(logDir, sparkConf, hadoopConf, outputBufferSize, |