diff options
author | Matei Zaharia <matei@eecs.berkeley.edu> | 2013-12-04 23:33:04 -0800 |
---|---|---|
committer | Matei Zaharia <matei@eecs.berkeley.edu> | 2013-12-04 23:33:04 -0800 |
commit | 72b696156c8662cae2cef4b943520b4be86148ea (patch) | |
tree | 14b4a2c841d78ddcf5accf047565286afc1ae0cd /core | |
parent | 182f9baeed8e4cc62ca14ae04413394477a7ccfb (diff) | |
parent | 46b87b8a258608bc85b98664f24cf59ef9ce6af6 (diff) | |
download | spark-72b696156c8662cae2cef4b943520b4be86148ea.tar.gz spark-72b696156c8662cae2cef4b943520b4be86148ea.tar.bz2 spark-72b696156c8662cae2cef4b943520b4be86148ea.zip |
Merge pull request #199 from harveyfeng/yarn-2.2
Hadoop 2.2 migration
Includes support for the YARN API stabilized in the Hadoop 2.2 release, and a few style patches.
Short description for each set of commits:
a98f5a0 - "Misc style changes in the 'yarn' package"
a67ebf4 - "A few more style fixes in the 'yarn' package"
Both of these are some minor style changes, such as fixing lines over 100 chars, to the existing YARN code.
ab8652f - "Add a 'new-yarn' directory ... "
Copies everything from `SPARK_HOME/yarn` to `SPARK_HOME/new-yarn`. No actual code changes here.
4f1c3fa - "Hadoop 2.2 YARN API migration ..."
API patches to code in the `SPARK_HOME/new-yarn` directory. There are a few more small style changes mixed in, too.
Based on @colorant's Hadoop 2.2 support for the scala-2.10 branch in #141.
a1a1c62 - "Add optional Hadoop 2.2 settings in sbt build ... "
If Spark should be built against Hadoop 2.2, then:
a) the `org.apache.spark.deploy.yarn` package will be compiled from the `new-yarn` directory.
b) Protobuf v2.5 will be used as a Spark dependency, since Hadoop 2.2 depends on it. Also, Spark will be built against a version of Akka v2.0.5 that's built against Protobuf 2.5, named `akka-2.0.5-protobuf-2.5`. The patched Akka is here: https://github.com/harveyfeng/akka/tree/2.0.5-protobuf-2.5, and was published to local Ivy during testing.
There's also a new boolean environment variable, `SPARK_IS_NEW_HADOOP`, that users can manually set if their `SPARK_HADOOP_VERSION` specification does not start with `2.2`, which is how the build file tries to detect a 2.2 version. Not sure if this is necessary or done in the best way, though...
Diffstat (limited to 'core')
-rw-r--r-- | core/pom.xml | 10 | ||||
-rw-r--r-- | core/src/main/scala/org/apache/spark/SparkContext.scala | 2 |
2 files changed, 4 insertions, 8 deletions
diff --git a/core/pom.xml b/core/pom.xml index 6af229c71d..38f4be1280 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -95,15 +95,15 @@ <version>0.3.1</version> </dependency> <dependency> - <groupId>com.typesafe.akka</groupId> + <groupId>${akka.group}</groupId> <artifactId>akka-actor</artifactId> </dependency> <dependency> - <groupId>com.typesafe.akka</groupId> + <groupId>${akka.group}</groupId> <artifactId>akka-remote</artifactId> </dependency> <dependency> - <groupId>com.typesafe.akka</groupId> + <groupId>${akka.group}</groupId> <artifactId>akka-slf4j</artifactId> </dependency> <dependency> @@ -127,10 +127,6 @@ <artifactId>colt</artifactId> </dependency> <dependency> - <groupId>com.github.scala-incubator.io</groupId> - <artifactId>scala-io-file_2.9.2</artifactId> - </dependency> - <dependency> <groupId>org.apache.mesos</groupId> <artifactId>mesos</artifactId> </dependency> diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala index 3da13133da..66006bf212 100644 --- a/core/src/main/scala/org/apache/spark/SparkContext.scala +++ b/core/src/main/scala/org/apache/spark/SparkContext.scala @@ -81,7 +81,7 @@ class SparkContext( val sparkHome: String = null, val jars: Seq[String] = Nil, val environment: Map[String, String] = Map(), - // This is used only by yarn for now, but should be relevant to other cluster types (mesos, etc) + // This is used only by YARN for now, but should be relevant to other cluster types (Mesos, etc) // too. This is typically generated from InputFormatInfo.computePreferredLocations .. host, set // of data-local splits on host val preferredNodeLocationData: scala.collection.Map[String, scala.collection.Set[SplitInfo]] = |