diff options
| author | Andrew Or <andrew@databricks.com> | 2014-11-06 17:18:49 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2014-11-06 17:18:58 -0800 |
| commit | 6508953a4b8622312c1f0ae4b4b4275b5a2c2bd6 (patch) | |
| tree | 7cf8e6a9cdd93e1e9b5e31fa0cc548a5976d1a63 /yarn/alpha | |
| parent | 9ea0fac0eafd7264a30f36c0d20863700245991f (diff) | |
| download | spark-6508953a4b8622312c1f0ae4b4b4275b5a2c2bd6.tar.gz spark-6508953a4b8622312c1f0ae4b4b4275b5a2c2bd6.tar.bz2 spark-6508953a4b8622312c1f0ae4b4b4275b5a2c2bd6.zip | |
[SPARK-3797] Minor addendum to Yarn shuffle service
I did not realize there was a `network.util.JavaUtils` when I wrote this code. This PR moves the `ByteBuffer` string conversion to the appropriate place. I tested the changes on a stable yarn cluster.
Author: Andrew Or <andrew@databricks.com>
Closes #3144 from andrewor14/yarn-shuffle-util and squashes the following commits:
b6c08bf [Andrew Or] Remove unused import
94e205c [Andrew Or] Use netty Unpooled
85202a5 [Andrew Or] Use guava Charsets
057135b [Andrew Or] Reword comment
adf186d [Andrew Or] Move byte buffer String conversion logic to JavaUtils
(cherry picked from commit 96136f222abd4f3abd10cb78a4ebecdb21f3bde7)
Signed-off-by: Andrew Or <andrew@databricks.com>
Diffstat (limited to 'yarn/alpha')
| -rw-r--r-- | yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala b/yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala index 5f47c79cab..7023a11706 100644 --- a/yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala +++ b/yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala @@ -36,7 +36,7 @@ import org.apache.hadoop.yarn.ipc.YarnRPC import org.apache.hadoop.yarn.util.{Apps, ConverterUtils, Records, ProtoUtils} import org.apache.spark.{SecurityManager, SparkConf, Logging} -import org.apache.spark.network.sasl.ShuffleSecretManager +import org.apache.spark.network.util.JavaUtils @deprecated("use yarn/stable", "1.2.0") class ExecutorRunnable( @@ -98,7 +98,8 @@ class ExecutorRunnable( val secretString = securityMgr.getSecretKey() val secretBytes = if (secretString != null) { - ShuffleSecretManager.stringToBytes(secretString) + // This conversion must match how the YarnShuffleService decodes our secret + JavaUtils.stringToBytes(secretString) } else { // Authentication is not enabled, so just provide dummy metadata ByteBuffer.allocate(0) |