diff options
author | Cheng Lian <lian@databricks.com> | 2015-08-17 17:25:14 -0700 |
---|---|---|
committer | Reynold Xin <rxin@databricks.com> | 2015-08-17 17:25:21 -0700 |
commit | 407175e824169a01762bdd27f704ac017d6d3e60 (patch) | |
tree | 4ff34df763199d3a592b7183909380a0f40b34c9 /pom.xml | |
parent | 0f1417b6f31e53dd78aae2a0a661d9ba32dce5b7 (diff) | |
download | spark-407175e824169a01762bdd27f704ac017d6d3e60.tar.gz spark-407175e824169a01762bdd27f704ac017d6d3e60.tar.bz2 spark-407175e824169a01762bdd27f704ac017d6d3e60.zip |
[SPARK-9974] [BUILD] [SQL] Makes sure com.twitter:parquet-hadoop-bundle:1.6.0 is in SBT assembly jar
PR #7967 enables Spark SQL to persist Parquet tables in Hive compatible format when possible. One of the consequence is that, we have to set input/output classes to `MapredParquetInputFormat`/`MapredParquetOutputFormat`, which rely on com.twitter:parquet-hadoop:1.6.0 bundled with Hive 1.2.1.
When loading such a table in Spark SQL, `o.a.h.h.ql.metadata.Table` first loads these input/output format classes, and thus classes in com.twitter:parquet-hadoop:1.6.0. However, the scope of this dependency is defined as "runtime", and is not packaged into Spark assembly jar. This results in a `ClassNotFoundException`.
This issue can be worked around by asking users to add parquet-hadoop 1.6.0 via the `--driver-class-path` option. However, considering Maven build is immune to this problem, I feel it can be confusing and inconvenient for users.
So this PR fixes this issue by changing scope of parquet-hadoop 1.6.0 to "compile".
Author: Cheng Lian <lian@databricks.com>
Closes #8198 from liancheng/spark-9974/bundle-parquet-1.6.0.
(cherry picked from commit 52ae952574f5d641a398dd185e09e5a79318c8a9)
Signed-off-by: Reynold Xin <rxin@databricks.com>
Diffstat (limited to 'pom.xml')
-rw-r--r-- | pom.xml | 2 |
1 files changed, 1 insertions, 1 deletions
@@ -1598,7 +1598,7 @@ <groupId>com.twitter</groupId> <artifactId>parquet-hadoop-bundle</artifactId> <version>${hive.parquet.version}</version> - <scope>runtime</scope> + <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.flume</groupId> |