diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2014-09-24 23:10:26 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-09-24 23:10:26 -0700 |
| commit | b8487713d3bf288a4f6fc149e6ee4cc8196d6e7d (patch) | |
| tree | 82c6871266f7f21344c59607815e89f43a3e7596 /yarn/pom.xml | |
| parent | 8ca4ecb6a56b96bae21b33e27f6abdb53676683a (diff) | |
| download | spark-b8487713d3bf288a4f6fc149e6ee4cc8196d6e7d.tar.gz spark-b8487713d3bf288a4f6fc149e6ee4cc8196d6e7d.tar.bz2 spark-b8487713d3bf288a4f6fc149e6ee4cc8196d6e7d.zip | |
[SPARK-2778] [yarn] Add yarn integration tests.
This patch adds a couple of, currently, very simple integration tests
to make sure both client and cluster modes are working. The tests don't
do much yet other than run a simple job, but the plan is to enhance
them after we get the framework in.
The cluster tests are noisy, so redirect all log output to a file
like other tests do. Copying the conf around sucks but it's less
work than messing with maven/sbt and having to clean up other
projects.
Note the test is only added for yarn-stable. The code compiles
against yarn-alpha but there are two issues I ran into that I
could not overcome:
- an old netty dependency kept creeping into the classpath and
causing akka to not work, when using sbt; the old netty was
correctly suppressed under maven.
- MiniYARNCluster kept failing to execute containers because it
did not create the NM's local dir itself; this is apparently
a known behavior, but I'm not sure how to work around it.
None of those issues are present with the stable Yarn.
Also, these tests are a little slow to run. Apparently Spark doesn't
yet tag tests (so that these could be isolated in a "slow" batch),
so this is something to keep in mind.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #2257 from vanzin/yarn-tests and squashes the following commits:
6d5b84e [Marcelo Vanzin] Fix wrong system property being set.
8b0933d [Marcelo Vanzin] Merge branch 'master' into yarn-tests
5c2b56f [Marcelo Vanzin] Use custom log4j conf for Yarn containers.
ec73f17 [Marcelo Vanzin] More review feedback.
67f5b02 [Marcelo Vanzin] Review feedback.
f01517c [Marcelo Vanzin] Review feedback.
68fbbbf [Marcelo Vanzin] Use older constructor available in older Hadoop releases.
d07ef9a [Marcelo Vanzin] Merge branch 'master' into yarn-tests
add8416 [Marcelo Vanzin] [SPARK-2778] [yarn] Add yarn integration tests.
Diffstat (limited to 'yarn/pom.xml')
| -rw-r--r-- | yarn/pom.xml | 3 |
1 files changed, 1 insertions, 2 deletions
diff --git a/yarn/pom.xml b/yarn/pom.xml index 815a736c2e..8a7035c85e 100644 --- a/yarn/pom.xml +++ b/yarn/pom.xml @@ -140,7 +140,6 @@ <configuration> <environmentVariables> <SPARK_HOME>${basedir}/../..</SPARK_HOME> - <SPARK_CLASSPATH>${spark.classpath}</SPARK_CLASSPATH> </environmentVariables> </configuration> </plugin> @@ -148,7 +147,7 @@ <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory> <testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory> - + <resources> <resource> <directory>../common/src/main/resources</directory> |