diff options
author | FavioVazquez <favio.vazquezp@gmail.com> | 2015-05-14 15:22:58 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2015-05-14 15:22:58 +0100 |
commit | 7fb715de6d90c3eb756935440f75b1de674f8ece (patch) | |
tree | 3ab15d2cb807da9de933716d293ab7653f6808fb /pom.xml | |
parent | c1080b6fddb22d84694da2453e46a03fbc041576 (diff) | |
download | spark-7fb715de6d90c3eb756935440f75b1de674f8ece.tar.gz spark-7fb715de6d90c3eb756935440f75b1de674f8ece.tar.bz2 spark-7fb715de6d90c3eb756935440f75b1de674f8ece.zip |
[SPARK-7249] Updated Hadoop dependencies due to inconsistency in the versions
Updated Hadoop dependencies due to inconsistency in the versions. Now the global properties are the ones used by the hadoop-2.2 profile, and the profile was set to empty but kept for backwards compatibility reasons.
Changes proposed by vanzin resulting from previous pull-request https://github.com/apache/spark/pull/5783 that did not fixed the problem correctly.
Please let me know if this is the correct way of doing this, the comments of vanzin are in the pull-request mentioned.
Author: FavioVazquez <favio.vazquezp@gmail.com>
Closes #5786 from FavioVazquez/update-hadoop-dependencies and squashes the following commits:
11670e5 [FavioVazquez] - Added missing instance of -Phadoop-2.2 in create-release.sh
379f50d [FavioVazquez] - Added instances of -Phadoop-2.2 in create-release.sh, run-tests, scalastyle and building-spark.md - Reconstructed docs to not ask users to rely on default behavior
3f9249d [FavioVazquez] Merge branch 'master' of https://github.com/apache/spark into update-hadoop-dependencies
31bdafa [FavioVazquez] - Added missing instances in -Phadoop-1 in create-release.sh, run-tests and in the building-spark documentation
cbb93e8 [FavioVazquez] - Added comment related to SPARK-3710 about hadoop-yarn-server-tests in Hadoop 2.2 that fails to pull some needed dependencies
83dc332 [FavioVazquez] - Cleaned up the main POM concerning the yarn profile - Erased hadoop-2.2 profile from yarn/pom.xml and its content was integrated into yarn/pom.xml
93f7624 [FavioVazquez] - Deleted unnecessary comments and <activation> tag on the YARN profile in the main POM
668d126 [FavioVazquez] - Moved <dependencies> <activation> and <properties> sections of the hadoop-2.2 profile in the YARN POM to the YARN profile in the root POM - Erased unnecessary hadoop-2.2 profile from the YARN POM
fda6a51 [FavioVazquez] - Updated hadoop1 releases in create-release.sh due to changes in the default hadoop version set - Erased unnecessary instance of -Dyarn.version=2.2.0 in create-release.sh - Prettify comment in yarn/pom.xml
0470587 [FavioVazquez] - Erased unnecessary instance of -Phadoop-2.2 -Dhadoop.version=2.2.0 in create-release.sh - Updated how the releases are made in the create-release.sh no that the default hadoop version is the 2.2.0 - Erased unnecessary instance of -Phadoop-2.2 -Dhadoop.version=2.2.0 in scalastyle - Erased unnecessary instance of -Phadoop-2.2 -Dhadoop.version=2.2.0 in run-tests - Better example given in the hadoop-third-party-distributions.md now that the default hadoop version is 2.2.0
a650779 [FavioVazquez] - Default value of avro.mapred.classifier has been set to hadoop2 in pom.xml - Cleaned up hadoop-2.3 and 2.4 profiles due to change in the default set in avro.mapred.classifier in pom.xml
199f40b [FavioVazquez] - Erased unnecessary CDH5-specific note in docs/building-spark.md - Remove example of instance -Phadoop-2.2 -Dhadoop.version=2.2.0 in docs/building-spark.md - Enabled hadoop-2.2 profile when the Hadoop version is 2.2.0, which is now the default .Added comment in the yarn/pom.xml to specify that.
88a8b88 [FavioVazquez] - Simplified Hadoop profiles due to new setting of global properties in the pom.xml file - Added comment to specify that the hadoop-2.2 profile is now the default hadoop profile in the pom.xml file - Erased hadoop-2.2 from related hadoop profiles now that is a no-op in the make-distribution.sh file
70b8344 [FavioVazquez] - Fixed typo in the make-distribution.sh file and added hadoop-1 in the Related profiles
287fa2f [FavioVazquez] - Updated documentation about specifying the hadoop version in building-spark. Now is clear that Spark will build against Hadoop 2.2.0 by default. - Added Cloudera CDH 5.3.3 without MapReduce example in the building-spark doc.
1354292 [FavioVazquez] - Fixed hadoop-1 version to match jenkins build profile in hadoop1.0 tests and documentation
6b4bfaf [FavioVazquez] - Cleanup in hadoop-2.x profiles since they contained mostly redundant stuff.
7e9955d [FavioVazquez] - Updated Hadoop dependencies due to inconsistency in the versions. Now the global properties are the ones used by the hadoop-2.2 profile, and the profile was set to empty but kept for backwards compatibility reasons
660decc [FavioVazquez] - Updated Hadoop dependencies due to inconsistency in the versions. Now the global properties are the ones used by the hadoop-2.2 profile, and the profile was set to empty but kept for backwards compatibility reasons
ec91ce3 [FavioVazquez] - Updated protobuf-java version of com.google.protobuf dependancy to fix blocking error when connecting to HDFS via the Hadoop Cloudera HDFS CDH5 (fix for 2.5.0-cdh5.3.3 version)
Diffstat (limited to 'pom.xml')
-rw-r--r-- | pom.xml | 33 |
1 files changed, 15 insertions, 18 deletions
@@ -122,9 +122,9 @@ <slf4j.version>1.7.10</slf4j.version> <log4j.version>1.2.17</log4j.version> <hadoop.version>2.2.0</hadoop.version> - <protobuf.version>2.4.1</protobuf.version> + <protobuf.version>2.5.0</protobuf.version> <yarn.version>${hadoop.version}</yarn.version> - <hbase.version>0.98.7-hadoop1</hbase.version> + <hbase.version>0.98.7-hadoop2</hbase.version> <hbase.artifact>hbase</hbase.artifact> <flume.version>1.4.0</flume.version> <zookeeper.version>3.4.5</zookeeper.version> @@ -143,7 +143,7 @@ <oro.version>2.0.8</oro.version> <codahale.metrics.version>3.1.0</codahale.metrics.version> <avro.version>1.7.7</avro.version> - <avro.mapred.classifier></avro.mapred.classifier> + <avro.mapred.classifier>hadoop2</avro.mapred.classifier> <jets3t.version>0.7.1</jets3t.version> <aws.java.sdk.version>1.8.3</aws.java.sdk.version> <aws.kinesis.client.version>1.1.0</aws.kinesis.client.version> @@ -155,7 +155,7 @@ <jline.version>${scala.version}</jline.version> <jline.groupid>org.scala-lang</jline.groupid> <jodd.version>3.6.3</jodd.version> - <codehaus.jackson.version>1.8.8</codehaus.jackson.version> + <codehaus.jackson.version>1.9.13</codehaus.jackson.version> <fasterxml.jackson.version>2.4.4</fasterxml.jackson.version> <snappy.version>1.1.1.7</snappy.version> <netlib.java.version>1.1.2</netlib.java.version> @@ -1644,26 +1644,27 @@ --> <profile> - <id>hadoop-2.2</id> + <id>hadoop-1</id> <properties> - <hadoop.version>2.2.0</hadoop.version> - <protobuf.version>2.5.0</protobuf.version> - <hbase.version>0.98.7-hadoop2</hbase.version> - <avro.mapred.classifier>hadoop2</avro.mapred.classifier> - <codehaus.jackson.version>1.9.13</codehaus.jackson.version> + <hadoop.version>1.0.4</hadoop.version> + <protobuf.version>2.4.1</protobuf.version> + <hbase.version>0.98.7-hadoop1</hbase.version> + <avro.mapred.classifier>hadoop1</avro.mapred.classifier> + <codehaus.jackson.version>1.8.8</codehaus.jackson.version> </properties> </profile> <profile> + <id>hadoop-2.2</id> + <!-- SPARK-7249: Default hadoop profile. Uses global properties. --> + </profile> + + <profile> <id>hadoop-2.3</id> <properties> <hadoop.version>2.3.0</hadoop.version> - <protobuf.version>2.5.0</protobuf.version> <jets3t.version>0.9.3</jets3t.version> - <hbase.version>0.98.7-hadoop2</hbase.version> <commons.math3.version>3.1.1</commons.math3.version> - <avro.mapred.classifier>hadoop2</avro.mapred.classifier> - <codehaus.jackson.version>1.9.13</codehaus.jackson.version> </properties> </profile> @@ -1671,12 +1672,8 @@ <id>hadoop-2.4</id> <properties> <hadoop.version>2.4.0</hadoop.version> - <protobuf.version>2.5.0</protobuf.version> <jets3t.version>0.9.3</jets3t.version> - <hbase.version>0.98.7-hadoop2</hbase.version> <commons.math3.version>3.1.1</commons.math3.version> - <avro.mapred.classifier>hadoop2</avro.mapred.classifier> - <codehaus.jackson.version>1.9.13</codehaus.jackson.version> </properties> </profile> |