diff options
| author | Dongjoon Hyun <dongjoon@apache.org> | 2017-02-08 12:21:49 +0000 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2017-02-08 12:21:49 +0000 |
| commit | 0077bfcb93832d93009f73f4b80f2e3d98fd2fa4 (patch) | |
| tree | 8aafe961fbd1ea1bb3951bc34838d97de162c716 | |
| parent | e8d3fca4502d5f5b8f38525b5fdabe80ccf9a8ec (diff) | |
| download | spark-0077bfcb93832d93009f73f4b80f2e3d98fd2fa4.tar.gz spark-0077bfcb93832d93009f73f4b80f2e3d98fd2fa4.tar.bz2 spark-0077bfcb93832d93009f73f4b80f2e3d98fd2fa4.zip | |
[SPARK-19409][BUILD][TEST-MAVEN] Fix ParquetAvroCompatibilitySuite failure due to test dependency on avro
## What changes were proposed in this pull request?
After using Apache Parquet 1.8.2, `ParquetAvroCompatibilitySuite` fails on **Maven** test. It is because `org.apache.parquet.avro.AvroParquetWriter` in the test code used new `avro 1.8.0` specific class, `LogicalType`. This PR aims to fix the test dependency of `sql/core` module to use avro 1.8.0.
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/2530/consoleFull
```
ParquetAvroCompatibilitySuite:
*** RUN ABORTED ***
java.lang.NoClassDefFoundError: org/apache/avro/LogicalType
at org.apache.parquet.avro.AvroParquetWriter.writeSupport(AvroParquetWriter.java:144)
```
## How was this patch tested?
Pass the existing test with **Maven**.
```
$ build/mvn -Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive -Phive-thriftserver test
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:07 h
[INFO] Finished at: 2017-02-04T05:41:43+00:00
[INFO] Final Memory: 77M/987M
[INFO] ------------------------------------------------------------------------
```
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #16795 from dongjoon-hyun/SPARK-19409-2.
| -rw-r--r-- | resource-managers/mesos/pom.xml | 7 | ||||
| -rw-r--r-- | sql/core/pom.xml | 13 |
2 files changed, 20 insertions, 0 deletions
diff --git a/resource-managers/mesos/pom.xml b/resource-managers/mesos/pom.xml index c0a8f9a344..03846d9f5a 100644 --- a/resource-managers/mesos/pom.xml +++ b/resource-managers/mesos/pom.xml @@ -49,6 +49,13 @@ </dependency> <dependency> + <groupId>org.apache.spark</groupId> + <artifactId>spark-tags_${scala.binary.version}</artifactId> + <type>test-jar</type> + <scope>test</scope> + </dependency> + + <dependency> <groupId>org.apache.mesos</groupId> <artifactId>mesos</artifactId> <version>${mesos.version}</version> diff --git a/sql/core/pom.xml b/sql/core/pom.xml index b8aa698090..69d797b479 100644 --- a/sql/core/pom.xml +++ b/sql/core/pom.xml @@ -129,6 +129,19 @@ <artifactId>parquet-avro</artifactId> <scope>test</scope> </dependency> + <!-- + This version of avro test-dep is different from the one defined + in the parent pom. The parent pom has avro 1.7.7 test-dep for Hadoop. + Here, ParquetAvroCompatibilitySuite uses parquet-avro's AvroParquetWriter + which uses avro 1.8.0+ specific API. In Maven 3, we need to have + this here to have different versions for the same artifact. + --> + <dependency> + <groupId>org.apache.avro</groupId> + <artifactId>avro</artifactId> + <version>1.8.1</version> + <scope>test</scope> + </dependency> <dependency> <groupId>org.mockito</groupId> <artifactId>mockito-core</artifactId> |