diff options
| author | Marcelo Vanzin <vanzin@cloudera.com> | 2014-09-12 14:54:42 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-09-12 14:54:42 -0700 |
| commit | af2583826c15d2a4e2732017ea20feeff0fb79f6 (patch) | |
| tree | 0d25121785197f37973e46c965c8471955761d56 /bin | |
| parent | 1d767967e925f1d727957c2d43383ef6ad2c5d5e (diff) | |
| download | spark-af2583826c15d2a4e2732017ea20feeff0fb79f6.tar.gz spark-af2583826c15d2a4e2732017ea20feeff0fb79f6.tar.bz2 spark-af2583826c15d2a4e2732017ea20feeff0fb79f6.zip | |
[SPARK-3217] Add Guava to classpath when SPARK_PREPEND_CLASSES is set.
When that option is used, the compiled classes from the build directory
are prepended to the classpath. Now that we avoid packaging Guava, that
means we have classes referencing the original Guava location in the app's
classpath, so errors happen.
For that case, add Guava manually to the classpath.
Note: if Spark is compiled with "-Phadoop-provided", it's tricky to
make things work with SPARK_PREPEND_CLASSES, because you need to add
the Hadoop classpath using SPARK_CLASSPATH and that means the older
Hadoop Guava overrides the newer one Spark needs. So someone using
SPARK_PREPEND_CLASSES needs to remember to not use that profile.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #2141 from vanzin/SPARK-3217 and squashes the following commits:
b967324 [Marcelo Vanzin] [SPARK-3217] Add Guava to classpath when SPARK_PREPEND_CLASSES is set.
Diffstat (limited to 'bin')
| -rwxr-xr-x | bin/compute-classpath.sh | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/bin/compute-classpath.sh b/bin/compute-classpath.sh index 15c6779402..0f63e36d8a 100755 --- a/bin/compute-classpath.sh +++ b/bin/compute-classpath.sh @@ -43,6 +43,7 @@ if [ -n "$SPARK_PREPEND_CLASSES" ]; then echo "NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark"\ "classes ahead of assembly." >&2 CLASSPATH="$CLASSPATH:$FWDIR/core/target/scala-$SCALA_VERSION/classes" + CLASSPATH="$CLASSPATH:$FWDIR/core/target/jars/*" CLASSPATH="$CLASSPATH:$FWDIR/repl/target/scala-$SCALA_VERSION/classes" CLASSPATH="$CLASSPATH:$FWDIR/mllib/target/scala-$SCALA_VERSION/classes" CLASSPATH="$CLASSPATH:$FWDIR/bagel/target/scala-$SCALA_VERSION/classes" |