diff options
author | Steve Loughran <stevel@hortonworks.com> | 2015-08-03 15:24:34 -0700 |
---|---|---|
committer | Michael Armbrust <michael@databricks.com> | 2015-08-03 15:25:20 -0700 |
commit | 6bd12e819451dbec602f1f2bbfc4c4bebc881e72 (patch) | |
tree | d0df37d141895084eb52875f324141de42108605 /pom.xml | |
parent | db5832708267f4a8413b0ad19c6a454c93f7800e (diff) | |
download | spark-6bd12e819451dbec602f1f2bbfc4c4bebc881e72.tar.gz spark-6bd12e819451dbec602f1f2bbfc4c4bebc881e72.tar.bz2 spark-6bd12e819451dbec602f1f2bbfc4c4bebc881e72.zip |
[SPARK-8064] [SQL] Build against Hive 1.2.1
Cherry picked the parts of the initial SPARK-8064 WiP branch needed to get sql/hive to compile against hive 1.2.1. That's the ASF release packaged under org.apache.hive, not any fork.
Tests not run yet: that's what the machines are for
Author: Steve Loughran <stevel@hortonworks.com>
Author: Cheng Lian <lian@databricks.com>
Author: Michael Armbrust <michael@databricks.com>
Author: Patrick Wendell <patrick@databricks.com>
Closes #7191 from steveloughran/stevel/feature/SPARK-8064-hive-1.2-002 and squashes the following commits:
7556d85 [Cheng Lian] Updates .q files and corresponding golden files
ef4af62 [Steve Loughran] Merge commit '6a92bb09f46a04d6cd8c41bdba3ecb727ebb9030' into stevel/feature/SPARK-8064-hive-1.2-002
6a92bb0 [Cheng Lian] Overrides HiveConf time vars
dcbb391 [Cheng Lian] Adds com.twitter:parquet-hadoop-bundle:1.6.0 for Hive Parquet SerDe
0bbe475 [Steve Loughran] SPARK-8064 scalastyle rejects the standard Hadoop ASF license header...
fdf759b [Steve Loughran] SPARK-8064 classpath dependency suite to be in sync with shading in final (?) hive-exec spark
7a6c727 [Steve Loughran] SPARK-8064 switch to second staging repo of the spark-hive artifacts. This one has the protobuf-shaded hive-exec jar
376c003 [Steve Loughran] SPARK-8064 purge duplicate protobuf declaration
2c74697 [Steve Loughran] SPARK-8064 switch to the protobuf shaded hive-exec jar with tests to chase it down
cc44020 [Steve Loughran] SPARK-8064 remove hadoop.version from runtest.py, as profile will fix that automatically.
6901fa9 [Steve Loughran] SPARK-8064 explicit protobuf import
da310dc [Michael Armbrust] Fixes for Hive tests.
a775a75 [Steve Loughran] SPARK-8064 cherry-pick-incomplete
7404f34 [Patrick Wendell] Add spark-hive staging repo
832c164 [Steve Loughran] SPARK-8064 try to supress compiler warnings on Complex.java pasted-thrift-code
312c0d4 [Steve Loughran] SPARK-8064 maven/ivy dependency purge; calcite declaration needed
fa5ae7b [Steve Loughran] HIVE-8064 fix up hive-thriftserver dependencies and cut back on evicted references in the hive- packages; this keeps mvn and ivy resolution compatible, as the reconciliation policy is "by hand"
c188048 [Steve Loughran] SPARK-8064 manage the Hive depencencies to that -things that aren't needed are excluded -sql/hive built with ivy is in sync with the maven reconciliation policy, rather than latest-first
4c8be8d [Cheng Lian] WIP: Partial fix for Thrift server and CLI tests
314eb3c [Steve Loughran] SPARK-8064 deprecation warning noise in one of the tests
17b0341 [Steve Loughran] SPARK-8064 IDE-hinted cleanups of Complex.java to reduce compiler warnings. It's all autogenerated code, so still ugly.
d029b92 [Steve Loughran] SPARK-8064 rely on unescaping to have already taken place, so go straight to map of serde options
23eca7e [Steve Loughran] HIVE-8064 handle raw and escaped property tokens
54d9b06 [Steve Loughran] SPARK-8064 fix compilation regression surfacing from rebase
0b12d5f [Steve Loughran] HIVE-8064 use subset of hive complex type whose types deserialize
fce73b6 [Steve Loughran] SPARK-8064 poms rely implicitly on the version of kryo chill provides
fd3aa5d [Steve Loughran] SPARK-8064 version of hive to d/l from ivy is 1.2.1
dc73ece [Steve Loughran] SPARK-8064 revert to master's determinstic pushdown strategy
d3c1e4a [Steve Loughran] SPARK-8064 purge UnionType
051cc21 [Steve Loughran] SPARK-8064 switch to an unshaded version of hive-exec-core, which must have been built with Kryo 2.21. This currently looks for a (locally built) version 1.2.1.spark
6684c60 [Steve Loughran] SPARK-8064 ignore RTE raised in blocking process.exitValue() call
e6121e5 [Steve Loughran] SPARK-8064 address review comments
aa43dc6 [Steve Loughran] SPARK-8064 more robust teardown on JavaMetastoreDatasourcesSuite
f2bff01 [Steve Loughran] SPARK-8064 better takeup of asynchronously caught error text
8b1ef38 [Steve Loughran] SPARK-8064: on failures executing spark-submit in HiveSparkSubmitSuite, print command line and all logged output.
5a9ce6b [Steve Loughran] SPARK-8064 add explicit reason for kv split failure, rather than array OOB. *does not address the issue*
642b63a [Steve Loughran] SPARK-8064 reinstate something cut briefly during rebasing
97194dc [Steve Loughran] SPARK-8064 add extra logging to the YarnClusterSuite classpath test. There should be no reason why this is failing on jenkins, but as it is (and presumably its CP-related), improve the logging including any exception raised.
335357f [Steve Loughran] SPARK-8064 fail fast on thrive process spawning tests on exit codes and/or error string patterns seen in log.
3ed872f [Steve Loughran] SPARK-8064 rename field double to dbl
bca55e5 [Steve Loughran] SPARK-8064 missed one of the `date` escapes
41d6479 [Steve Loughran] SPARK-8064 wrap tests with withTable() calls to avoid table-exists exceptions
2bc29a4 [Steve Loughran] SPARK-8064 ParquetSuites to escape `date` field name
1ab9bc4 [Steve Loughran] SPARK-8064 TestHive to use sered2.thrift.test.Complex
bf3a249 [Steve Loughran] SPARK-8064: more resubmit than fix; tighten startup timeout to 60s. Still no obvious reason why jersey server code in spark-assembly isn't being picked up -it hasn't been shaded
c829b8f [Steve Loughran] SPARK-8064: reinstate yarn-rm-server dependencies to hive-exec to ensure that jersey server is on classpath on hadoop versions < 2.6
0b0f738 [Steve Loughran] SPARK-8064: thrift server startup to fail fast on any exception in the main thread
13abaf1 [Steve Loughran] SPARK-8064 Hive compatibilty tests sin sync with explain/show output from Hive 1.2.1
d14d5ea [Steve Loughran] SPARK-8064: DATE is now a predicate; you can't use it as a field in select ops
26eef1c [Steve Loughran] SPARK-8064: HIVE-9039 renamed TOK_UNION => TOK_UNIONALL while adding TOK_UNIONDISTINCT
3d64523 [Steve Loughran] SPARK-8064 improve diagns on uknown token; fix scalastyle failure
d0360f6 [Steve Loughran] SPARK-8064: delicate merge in of the branch vanzin/hive-1.1
1126e5a [Steve Loughran] SPARK-8064: name of unrecognized file format wasn't appearing in error text
8cb09c4 [Steve Loughran] SPARK-8064: test resilience/assertion improvements. Independent of the rest of the work; can be backported to earlier versions
dec12cb [Steve Loughran] SPARK-8064: when a CLI suite test fails include the full output text in the raised exception; this ensures that the stdout/stderr is included in jenkins reports, so it becomes possible to diagnose the cause.
463a670 [Steve Loughran] SPARK-8064 run-tests.py adds a hadoop-2.6 profile, and changes info messages to say "w/Hive 1.2.1" in console output
2531099 [Steve Loughran] SPARK-8064 successful attempt to get rid of pentaho as a transitive dependency of hive-exec
1d59100 [Steve Loughran] SPARK-8064 (unsuccessful) attempt to get rid of pentaho as a transitive dependency of hive-exec
75733fc [Steve Loughran] SPARK-8064 change thrift binary startup message to "Starting ThriftBinaryCLIService on port"
3ebc279 [Steve Loughran] SPARK-8064 move strings used to check for http/bin thrift services up into constants
c80979d [Steve Loughran] SPARK-8064: SparkSQLCLIDriver drops remote mode support. CLISuite Tests pass instead of timing out: undetected regression?
27e8370 [Steve Loughran] SPARK-8064 fix some style & IDE warnings
00e50d6 [Steve Loughran] SPARK-8064 stop excluding hive shims from dependency (commented out , for now)
cb4f142 [Steve Loughran] SPARK-8054 cut pentaho dependency from calcite
f7aa9cb [Steve Loughran] SPARK-8064 everything compiles with some commenting and moving of classes into a hive package
6c310b4 [Steve Loughran] SPARK-8064 subclass Hive ServerOptionsProcessor to make it public again
f61a675 [Steve Loughran] SPARK-8064 thrift server switched to Hive 1.2.1, though it doesn't compile everywhere
4890b9d [Steve Loughran] SPARK-8064, build against Hive 1.2.1
(cherry picked from commit a2409d1c8e8ddec04b529ac6f6a12b5993f0eeda)
Signed-off-by: Michael Armbrust <michael@databricks.com>
Diffstat (limited to 'pom.xml')
-rw-r--r-- | pom.xml | 654 |
1 files changed, 642 insertions, 12 deletions
@@ -134,11 +134,12 @@ <curator.version>2.4.0</curator.version> <hive.group>org.spark-project.hive</hive.group> <!-- Version used in Maven Hive dependency --> - <hive.version>0.13.1a</hive.version> + <hive.version>1.2.1.spark</hive.version> <!-- Version used for internal directory structure --> - <hive.version.short>0.13.1</hive.version.short> + <hive.version.short>1.2.1</hive.version.short> <derby.version>10.10.1.1</derby.version> <parquet.version>1.7.0</parquet.version> + <hive.parquet.version>1.6.0</hive.parquet.version> <jblas.version>1.2.4</jblas.version> <jetty.version>8.1.14.v20131031</jetty.version> <orbit.version>3.0.0.v201112011016</orbit.version> @@ -151,7 +152,10 @@ <jets3t.version>0.7.1</jets3t.version> <aws.java.sdk.version>1.9.16</aws.java.sdk.version> <aws.kinesis.client.version>1.2.1</aws.kinesis.client.version> + <!-- org.apache.httpcomponents/httpclient--> <commons.httpclient.version>4.3.2</commons.httpclient.version> + <!-- commons-httpclient/commons-httpclient--> + <httpclient.classic.version>3.1</httpclient.classic.version> <commons.math3.version>3.4.1</commons.math3.version> <scala.version>2.10.4</scala.version> <scala.binary.version>2.10</scala.binary.version> @@ -161,6 +165,23 @@ <fasterxml.jackson.version>2.4.4</fasterxml.jackson.version> <snappy.version>1.1.1.7</snappy.version> <netlib.java.version>1.1.2</netlib.java.version> + <calcite.version>1.2.0-incubating</calcite.version> + <commons-codec.version>1.10</commons-codec.version> + <!-- org.apache.commons/commons-lang/--> + <commons-lang2.version>2.6</commons-lang2.version> + <!-- org.apache.commons/commons-lang3/--> + <commons-lang3.version>3.3.2</commons-lang3.version> + <datanucleus-core.version>3.2.10</datanucleus-core.version> + <janino.version>2.7.8</janino.version> + <jersey.version>1.9</jersey.version> + <joda.version>2.5</joda.version> + <jodd.version>3.5.2</jodd.version> + <jsr305.version>1.3.9</jsr305.version> + <libthrift.version>0.9.2</libthrift.version> + + <!-- For maven shade plugin (see SPARK-8819) --> + <create.dependency.reduced.pom>false</create.dependency.reduced.pom> + <test.java.home>${java.home}</test.java.home> <!-- @@ -188,7 +209,6 @@ <MaxPermGen>512m</MaxPermGen> <CodeCacheSize>512m</CodeCacheSize> </properties> - <repositories> <repository> <id>central</id> @@ -247,6 +267,14 @@ </snapshots> </repository> <repository> + <id>spark-hive-staging</id> + <name>Staging Repo for Hive 1.2.1 (Spark Version)</name> + <url>https://oss.sonatype.org/content/repositories/orgspark-project-1113</url> + <releases> + <enabled>true</enabled> + </releases> + </repository> + <repository> <id>mapr-repo</id> <name>MapR Repository</name> <url>http://repository.mapr.com/maven/</url> @@ -257,12 +285,13 @@ <enabled>false</enabled> </snapshots> </repository> + <!-- returning unauthorized on some operations --> <repository> <id>spring-releases</id> <name>Spring Release Repository</name> <url>https://repo.spring.io/libs-release</url> <releases> - <enabled>true</enabled> + <enabled>false</enabled> </releases> <snapshots> <enabled>false</enabled> @@ -402,12 +431,17 @@ <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-lang3</artifactId> - <version>3.3.2</version> + <version>${commons-lang3.version}</version> + </dependency> + <dependency> + <groupId>org.apache.commons</groupId> + <artifactId>commons-lang</artifactId> + <version>${commons-lang2.version}</version> </dependency> <dependency> <groupId>commons-codec</groupId> <artifactId>commons-codec</artifactId> - <version>1.10</version> + <version>${commons-codec.version}</version> </dependency> <dependency> <groupId>org.apache.commons</groupId> @@ -422,7 +456,12 @@ <dependency> <groupId>com.google.code.findbugs</groupId> <artifactId>jsr305</artifactId> - <version>1.3.9</version> + <version>${jsr305.version}</version> + </dependency> + <dependency> + <groupId>commons-httpclient</groupId> + <artifactId>commons-httpclient</artifactId> + <version>${httpclient.classic.version}</version> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> @@ -439,6 +478,16 @@ <artifactId>selenium-java</artifactId> <version>2.42.2</version> <scope>test</scope> + <exclusions> + <exclusion> + <groupId>com.google.guava</groupId> + <artifactId>guava</artifactId> + </exclusion> + <exclusion> + <groupId>io.netty</groupId> + <artifactId>netty</artifactId> + </exclusion> + </exclusions> </dependency> <!-- Added for selenium only, and should match its dependent version: --> <dependency> @@ -624,16 +673,27 @@ <dependency> <groupId>com.sun.jersey</groupId> <artifactId>jersey-server</artifactId> - <version>1.9</version> + <version>${jersey.version}</version> <scope>${hadoop.deps.scope}</scope> </dependency> <dependency> <groupId>com.sun.jersey</groupId> <artifactId>jersey-core</artifactId> - <version>1.9</version> + <version>${jersey.version}</version> <scope>${hadoop.deps.scope}</scope> </dependency> <dependency> + <groupId>com.sun.jersey</groupId> + <artifactId>jersey-json</artifactId> + <version>${jersey.version}</version> + <exclusions> + <exclusion> + <groupId>stax</groupId> + <artifactId>stax-api</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-compiler</artifactId> <version>${scala.version}</version> @@ -1022,45 +1082,357 @@ <artifactId>hive-beeline</artifactId> <version>${hive.version}</version> <scope>${hive.deps.scope}</scope> + <exclusions> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-common</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-exec</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-jdbc</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-metastore</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-service</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-shims</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libthrift</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + </exclusion> + <exclusion> + <groupId>log4j</groupId> + <artifactId>log4j</artifactId> + </exclusion> + <exclusion> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + </exclusion> + </exclusions> </dependency> <dependency> <groupId>${hive.group}</groupId> <artifactId>hive-cli</artifactId> <version>${hive.version}</version> <scope>${hive.deps.scope}</scope> + <exclusions> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-common</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-exec</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-jdbc</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-metastore</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-serde</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-service</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-shims</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libthrift</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + </exclusion> + <exclusion> + <groupId>log4j</groupId> + <artifactId>log4j</artifactId> + </exclusion> + <exclusion> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + </exclusion> + </exclusions> </dependency> <dependency> <groupId>${hive.group}</groupId> - <artifactId>hive-exec</artifactId> + <artifactId>hive-common</artifactId> <version>${hive.version}</version> <scope>${hive.deps.scope}</scope> <exclusions> <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-shims</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.ant</groupId> + <artifactId>ant</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.zookeeper</groupId> + <artifactId>zookeeper</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + </exclusion> + <exclusion> + <groupId>log4j</groupId> + <artifactId>log4j</artifactId> + </exclusion> + <exclusion> <groupId>commons-logging</groupId> <artifactId>commons-logging</artifactId> </exclusion> + </exclusions> + </dependency> + + <dependency> + <groupId>${hive.group}</groupId> + <artifactId>hive-exec</artifactId> +<!-- + <classifier>core</classifier> +--> + <version>${hive.version}</version> + <scope>${hive.deps.scope}</scope> + <exclusions> + + <!-- pull this in when needed; the explicit definition culls the surplis--> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-metastore</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-shims</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-ant</artifactId> + </exclusion> + <!-- break the loop --> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>spark-client</artifactId> + </exclusion> + + <!-- excluded dependencies & transitive. + Some may be needed to be explicitly included--> + <exclusion> + <groupId>ant</groupId> + <artifactId>ant</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.ant</groupId> + <artifactId>ant</artifactId> + </exclusion> <exclusion> <groupId>com.esotericsoftware.kryo</groupId> <artifactId>kryo</artifactId> </exclusion> <exclusion> + <groupId>commons-codec</groupId> + <artifactId>commons-codec</artifactId> + </exclusion> + <exclusion> + <groupId>commons-httpclient</groupId> + <artifactId>commons-httpclient</artifactId> + </exclusion> + <exclusion> <groupId>org.apache.avro</groupId> <artifactId>avro-mapred</artifactId> </exclusion> + <!-- this is needed and must be explicitly included later--> + <exclusion> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-core</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.curator</groupId> + <artifactId>apache-curator</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.curator</groupId> + <artifactId>curator-client</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.curator</groupId> + <artifactId>curator-framework</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libthrift</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libfb303</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.zookeeper</groupId> + <artifactId>zookeeper</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + </exclusion> + <exclusion> + <groupId>log4j</groupId> + <artifactId>log4j</artifactId> + </exclusion> + <exclusion> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + </exclusion> </exclusions> </dependency> <dependency> <groupId>${hive.group}</groupId> <artifactId>hive-jdbc</artifactId> <version>${hive.version}</version> - <scope>${hive.deps.scope}</scope> + <exclusions> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-common</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-common</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-metastore</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-serde</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-service</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-shims</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.httpcomponents</groupId> + <artifactId>httpclient</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.httpcomponents</groupId> + <artifactId>httpcore</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.curator</groupId> + <artifactId>curator-framework</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libthrift</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libfb303</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.zookeeper</groupId> + <artifactId>zookeeper</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + </exclusion> + <exclusion> + <groupId>log4j</groupId> + <artifactId>log4j</artifactId> + </exclusion> + <exclusion> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + </exclusion> + </exclusions> </dependency> + <dependency> <groupId>${hive.group}</groupId> <artifactId>hive-metastore</artifactId> <version>${hive.version}</version> <scope>${hive.deps.scope}</scope> + <exclusions> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-serde</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-shims</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libfb303</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libthrift</artifactId> + </exclusion> + <exclusion> + <groupId>com.google.guava</groupId> + <artifactId>guava</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + </exclusion> + </exclusions> </dependency> + <dependency> <groupId>${hive.group}</groupId> <artifactId>hive-serde</artifactId> @@ -1068,12 +1440,141 @@ <scope>${hive.deps.scope}</scope> <exclusions> <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-common</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-shims</artifactId> + </exclusion> + <exclusion> + <groupId>commons-codec</groupId> + <artifactId>commons-codec</artifactId> + </exclusion> + <exclusion> + <groupId>com.google.code.findbugs</groupId> + <artifactId>jsr305</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.avro</groupId> + <artifactId>avro</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libthrift</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libfb303</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + </exclusion> + <exclusion> + <groupId>log4j</groupId> + <artifactId>log4j</artifactId> + </exclusion> + <exclusion> <groupId>commons-logging</groupId> <artifactId>commons-logging</artifactId> </exclusion> + </exclusions> + </dependency> + + <dependency> + <groupId>${hive.group}</groupId> + <artifactId>hive-service</artifactId> + <version>${hive.version}</version> + <scope>${hive.deps.scope}</scope> + <exclusions> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-common</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-exec</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-metastore</artifactId> + </exclusion> + <exclusion> + <groupId>${hive.group}</groupId> + <artifactId>hive-shims</artifactId> + </exclusion> + <exclusion> + <groupId>commons-codec</groupId> + <artifactId>commons-codec</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.curator</groupId> + <artifactId>curator-framework</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.curator</groupId> + <artifactId>curator-recipes</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libfb303</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libthrift</artifactId> + </exclusion> + </exclusions> + </dependency> + + <!-- hive shims pulls in hive 0.23 and a transitive dependency of the Hadoop version + Hive was built against. This dependency cuts out the YARN/hadoop dependency, which + is needed by Hive to submit work to a YARN cluster.--> + <dependency> + <groupId>${hive.group}</groupId> + <artifactId>hive-shims</artifactId> + <version>${hive.version}</version> + <scope>${hive.deps.scope}</scope> + <exclusions> + <exclusion> + <groupId>com.google.guava</groupId> + <artifactId>guava</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-yarn-server-resourcemanager</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.curator</groupId> + <artifactId>curator-framework</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.thrift</groupId> + <artifactId>libthrift</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.zookeeper</groupId> + <artifactId>zookeeper</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + </exclusion> + <exclusion> + <groupId>log4j</groupId> + <artifactId>log4j</artifactId> + </exclusion> <exclusion> <groupId>commons-logging</groupId> - <artifactId>commons-logging-api</artifactId> + <artifactId>commons-logging</artifactId> </exclusion> </exclusions> </dependency> @@ -1096,6 +1597,12 @@ <scope>${parquet.test.deps.scope}</scope> </dependency> <dependency> + <groupId>com.twitter</groupId> + <artifactId>parquet-hadoop-bundle</artifactId> + <version>${hive.parquet.version}</version> + <scope>runtime</scope> + </dependency> + <dependency> <groupId>org.apache.flume</groupId> <artifactId>flume-ng-core</artifactId> <version>${flume.version}</version> @@ -1135,6 +1642,125 @@ </exclusion> </exclusions> </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-core</artifactId> + <version>${calcite.version}</version> + <exclusions> + <exclusion> + <groupId>com.fasterxml.jackson.core</groupId> + <artifactId>jackson-annotations</artifactId> + </exclusion> + <exclusion> + <groupId>com.fasterxml.jackson.core</groupId> + <artifactId>jackson-core</artifactId> + </exclusion> + <exclusion> + <groupId>com.fasterxml.jackson.core</groupId> + <artifactId>jackson-databind</artifactId> + </exclusion> + <exclusion> + <groupId>com.google.guava</groupId> + <artifactId>guava</artifactId> + </exclusion> + <exclusion> + <groupId>com.google.code.findbugs</groupId> + <artifactId>jsr305</artifactId> + </exclusion> + <exclusion> + <groupId>org.codehaus.janino</groupId> + <artifactId>janino</artifactId> + </exclusion> + <!-- hsqldb interferes with the use of derby as the default db + in hive's use of datanucleus. + --> + <exclusion> + <groupId>org.hsqldb</groupId> + <artifactId>hsqldb</artifactId> + </exclusion> + <exclusion> + <groupId>org.pentaho</groupId> + <artifactId>pentaho-aggdesigner-algorithm</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-avatica</artifactId> + <version>${calcite.version}</version> + <exclusions> + <exclusion> + <groupId>com.fasterxml.jackson.core</groupId> + <artifactId>jackson-annotations</artifactId> + </exclusion> + <exclusion> + <groupId>com.fasterxml.jackson.core</groupId> + <artifactId>jackson-core</artifactId> + </exclusion> + <exclusion> + <groupId>com.fasterxml.jackson.core</groupId> + <artifactId>jackson-databind</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.codehaus.janino</groupId> + <artifactId>janino</artifactId> + <version>${janino.version}</version> + </dependency> + <dependency> + <groupId>joda-time</groupId> + <artifactId>joda-time</artifactId> + <version>${joda.version}</version> + </dependency> + <dependency> + <groupId>org.jodd</groupId> + <artifactId>jodd-core</artifactId> + <version>${jodd.version}</version> + </dependency> + <dependency> + <groupId>org.datanucleus</groupId> + <artifactId>datanucleus-core</artifactId> + <version>${datanucleus-core.version}</version> + </dependency> + <dependency> + <groupId>org.apache.thrift</groupId> + <artifactId>libthrift</artifactId> + <version>${libthrift.version}</version> + <exclusions> + <exclusion> + <groupId>org.apache.httpcomponents</groupId> + <artifactId>httpclient</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.httpcomponents</groupId> + <artifactId>httpcore</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.thrift</groupId> + <artifactId>libfb303</artifactId> + <version>${libthrift.version}</version> + <exclusions> + <exclusion> + <groupId>org.apache.httpcomponents</groupId> + <artifactId>httpclient</artifactId> + </exclusion> + <exclusion> + <groupId>org.apache.httpcomponents</groupId> + <artifactId>httpcore</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + </exclusion> + </exclusions> + </dependency> </dependencies> </dependencyManagement> @@ -1271,6 +1897,8 @@ <spark.ui.showConsoleProgress>false</spark.ui.showConsoleProgress> <spark.driver.allowMultipleContexts>true</spark.driver.allowMultipleContexts> <spark.unsafe.exceptionOnMemoryLeak>true</spark.unsafe.exceptionOnMemoryLeak> + <!-- Needed by sql/hive tests. --> + <test.src.tables>src</test.src.tables> </systemProperties> <failIfNoTests>false</failIfNoTests> </configuration> @@ -1305,6 +1933,8 @@ <spark.ui.showConsoleProgress>false</spark.ui.showConsoleProgress> <spark.driver.allowMultipleContexts>true</spark.driver.allowMultipleContexts> <spark.unsafe.exceptionOnMemoryLeak>true</spark.unsafe.exceptionOnMemoryLeak> + <!-- Needed by sql/hive tests. --> + <test.src.tables>__not_used__</test.src.tables> </systemProperties> </configuration> <executions> |