diff options
| author | xiaojian.fxj <xiaojian.fxj@alibaba-inc.com> | 2017-01-15 11:12:59 +0000 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2017-01-15 11:12:59 +0000 |
| commit | c9d612f82c290fc955cae93150ca5c5d74f12217 (patch) | |
| tree | 2beb23a4e5ed5288c89959ff3a7f49371c808d8b | |
| parent | 9112f31bb88bf68fa113dad5eeb5d748979ba339 (diff) | |
| download | spark-c9d612f82c290fc955cae93150ca5c5d74f12217.tar.gz spark-c9d612f82c290fc955cae93150ca5c5d74f12217.tar.bz2 spark-c9d612f82c290fc955cae93150ca5c5d74f12217.zip | |
[SPARK-19042] spark executor can't download the jars when uber jar's http url contains any query strings
If the uber jars' https contains any query strings, the Executor.updateDependencies method can't can't download the jars correctly. This is because the "localName = name.split("/").last" won't get the expected jar's url. The bug fix is the same as [SPARK-17855]
Author: xiaojian.fxj <xiaojian.fxj@alibaba-inc.com>
Closes #16509 from hustfxj/bug.
| -rw-r--r-- | core/src/main/scala/org/apache/spark/executor/Executor.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/core/src/main/scala/org/apache/spark/executor/Executor.scala b/core/src/main/scala/org/apache/spark/executor/Executor.scala index b6c0f0c460..db5d0d85ce 100644 --- a/core/src/main/scala/org/apache/spark/executor/Executor.scala +++ b/core/src/main/scala/org/apache/spark/executor/Executor.scala @@ -19,7 +19,7 @@ package org.apache.spark.executor import java.io.{File, NotSerializableException} import java.lang.management.ManagementFactory -import java.net.URL +import java.net.{URI, URL} import java.nio.ByteBuffer import java.util.Properties import java.util.concurrent.{ConcurrentHashMap, TimeUnit} @@ -640,7 +640,7 @@ private[spark] class Executor( currentFiles(name) = timestamp } for ((name, timestamp) <- newJars) { - val localName = name.split("/").last + val localName = new URI(name).getPath.split("/").last val currentTimeStamp = currentJars.get(name) .orElse(currentJars.get(localName)) .getOrElse(-1L) |