diff options
author | Patrick Wendell <pwendell@gmail.com> | 2013-09-10 21:38:22 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2013-09-10 22:05:13 -0700 |
commit | 0c1985b153a2dc2c891ae61c1ee67506926384ae (patch) | |
tree | 7ef83d89054315a412480dabe3f9079e58a29634 | |
parent | 2425eb85ca709273c48958f81a81c8a04657ea1f (diff) | |
download | spark-0c1985b153a2dc2c891ae61c1ee67506926384ae.tar.gz spark-0c1985b153a2dc2c891ae61c1ee67506926384ae.tar.bz2 spark-0c1985b153a2dc2c891ae61c1ee67506926384ae.zip |
Fix HDFS access bug with assembly build.
Due to this change in HDFS:
https://issues.apache.org/jira/browse/HADOOP-7549
there is a bug when using the new assembly builds. The symptom is that any HDFS access
results in an exception saying "No filesystem for scheme 'hdfs'". This adds a merge
strategy in the assembly build which fixes the problem.
-rw-r--r-- | assembly/pom.xml | 6 | ||||
-rw-r--r-- | project/SparkBuild.scala | 1 |
2 files changed, 7 insertions, 0 deletions
diff --git a/assembly/pom.xml b/assembly/pom.xml index d19f44d292..808a829e19 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -106,6 +106,12 @@ <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/> <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> + <resource>META-INF/services/org.apache.hadoop.fs.FileSystem</resource> + </transformer> + </transformers> + <transformers> + <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/> + <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> <resource>reference.conf</resource> </transformer> </transformers> diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala index a60b553b5a..0736ec3ab7 100644 --- a/project/SparkBuild.scala +++ b/project/SparkBuild.scala @@ -303,6 +303,7 @@ object SparkBuild extends Build { mergeStrategy in assembly := { case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard case m if m.toLowerCase.matches("meta-inf.*\\.sf$") => MergeStrategy.discard + case "META-INF/services/org.apache.hadoop.fs.FileSystem" => MergeStrategy.concat case "reference.conf" => MergeStrategy.concat case _ => MergeStrategy.first } |