diff options
author | hyukjinkwon <gurwls223@gmail.com> | 2017-04-18 11:05:00 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2017-04-18 11:05:00 +0100 |
commit | d4f10cbbe1b9d13e43d80a50d204781e1c5c2da9 (patch) | |
tree | 7c2f002fab9ee608730f12ae04210946531855a5 /project | |
parent | 07fd94e0d05e827fae65d6e0e1cb89e28c8f2771 (diff) | |
download | spark-d4f10cbbe1b9d13e43d80a50d204781e1c5c2da9.tar.gz spark-d4f10cbbe1b9d13e43d80a50d204781e1c5c2da9.tar.bz2 spark-d4f10cbbe1b9d13e43d80a50d204781e1c5c2da9.zip |
[SPARK-20343][BUILD] Force Avro 1.7.7 in sbt build to resolve build failure in SBT Hadoop 2.6 master on Jenkins
## What changes were proposed in this pull request?
This PR proposes to force Avro's version to 1.7.7 in core to resolve the build failure as below:
```
[error] /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData
[error] writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema))
[error]
```
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/2770/consoleFull
Note that this is a hack and should be removed in the future.
## How was this patch tested?
I only tested this actually overrides the dependency.
I tried many ways but I was unable to reproduce this in my local. Sean also tried the way I did but he was also unable to reproduce this.
Please refer the comments in https://github.com/apache/spark/pull/17477#issuecomment-294094092
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #17651 from HyukjinKwon/SPARK-20343-sbt.
Diffstat (limited to 'project')
-rw-r--r-- | project/SparkBuild.scala | 14 |
1 files changed, 12 insertions, 2 deletions
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala index e52baf51ae..77dae289f7 100644 --- a/project/SparkBuild.scala +++ b/project/SparkBuild.scala @@ -318,8 +318,8 @@ object SparkBuild extends PomBuild { enable(MimaBuild.mimaSettings(sparkHome, x))(x) } - /* Generate and pick the spark build info from extra-resources */ - enable(Core.settings)(core) + /* Generate and pick the spark build info from extra-resources and override a dependency */ + enable(Core.settings ++ CoreDependencyOverrides.settings)(core) /* Unsafe settings */ enable(Unsafe.settings)(unsafe) @@ -444,6 +444,16 @@ object DockerIntegrationTests { } /** + * Overrides to work around sbt's dependency resolution being different from Maven's in Unidoc. + * + * Note that, this is a hack that should be removed in the future. See SPARK-20343 + */ +object CoreDependencyOverrides { + lazy val settings = Seq( + dependencyOverrides += "org.apache.avro" % "avro" % "1.7.7") +} + +/** * Overrides to work around sbt's dependency resolution being different from Maven's. */ object DependencyOverrides { |