diff options
| author | Dennis Huo <dhuo@google.com> | 2015-07-21 13:12:11 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2015-07-21 13:12:11 -0700 |
| commit | c07838b5a9cdf96c0f49055ea1c397e0f0e915d2 (patch) | |
| tree | 7f46c9a055b7cb6fc795dfe257ea72d51161487d | |
| parent | 60c0ce134d90ef18852ed2c637d2f240b7f99ab9 (diff) | |
| download | spark-c07838b5a9cdf96c0f49055ea1c397e0f0e915d2.tar.gz spark-c07838b5a9cdf96c0f49055ea1c397e0f0e915d2.tar.bz2 spark-c07838b5a9cdf96c0f49055ea1c397e0f0e915d2.zip | |
[SPARK-9206] [SQL] Fix HiveContext classloading for GCS connector.
IsolatedClientLoader.isSharedClass includes all of com.google.\*, presumably
for Guava, protobuf, and/or other shared Google libraries, but needs to
count com.google.cloud.\* as "hive classes" when determining which ClassLoader
to use. Otherwise, things like HiveContext.parquetFile will throw a
ClassCastException when fs.defaultFS is set to a Google Cloud Storage (gs://)
path. On StackOverflow: http://stackoverflow.com/questions/31478955
EDIT: Adding yhuai who worked on the relevant classloading isolation pieces.
Author: Dennis Huo <dhuo@google.com>
Closes #7549 from dennishuo/dhuo-fix-hivecontext-gcs and squashes the following commits:
1f8db07 [Dennis Huo] Fix HiveContext classloading for GCS connector.
| -rw-r--r-- | sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala index 3d609a66f3..97fb981999 100644 --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala @@ -125,7 +125,7 @@ private[hive] class IsolatedClientLoader( name.contains("log4j") || name.startsWith("org.apache.spark.") || name.startsWith("scala.") || - name.startsWith("com.google") || + (name.startsWith("com.google") && !name.startsWith("com.google.cloud")) || name.startsWith("java.lang.") || name.startsWith("java.net") || sharedPrefixes.exists(name.startsWith) |