diff options
| author | Steven She <steven@canopylabs.com> | 2015-06-29 18:50:09 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2015-06-29 18:50:09 -0700 |
| commit | 4915e9e3bffb57eac319ef2173b4a6ae4073d25e (patch) | |
| tree | f49bef4177c5db7c4bdfb92de2d7856fd3b1a38c /sql/core | |
| parent | ecacb1e88a135c802e253793e7c863d6ca8d2408 (diff) | |
| download | spark-4915e9e3bffb57eac319ef2173b4a6ae4073d25e.tar.gz spark-4915e9e3bffb57eac319ef2173b4a6ae4073d25e.tar.bz2 spark-4915e9e3bffb57eac319ef2173b4a6ae4073d25e.zip | |
[SPARK-8669] [SQL] Fix crash with BINARY (ENUM) fields with Parquet 1.7
Patch to fix crash with BINARY fields with ENUM original types.
Author: Steven She <steven@canopylabs.com>
Closes #7048 from stevencanopy/SPARK-8669 and squashes the following commits:
2e72979 [Steven She] [SPARK-8669] [SQL] Fix crash with BINARY (ENUM) fields with Parquet 1.7
Diffstat (limited to 'sql/core')
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/parquet/CatalystSchemaConverter.scala | 2 | ||||
| -rw-r--r-- | sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetSchemaSuite.scala | 8 |
2 files changed, 9 insertions, 1 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/parquet/CatalystSchemaConverter.scala b/sql/core/src/main/scala/org/apache/spark/sql/parquet/CatalystSchemaConverter.scala index 4fd3e93b70..2be7c64612 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/parquet/CatalystSchemaConverter.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/parquet/CatalystSchemaConverter.scala @@ -177,7 +177,7 @@ private[parquet] class CatalystSchemaConverter( case BINARY => field.getOriginalType match { - case UTF8 => StringType + case UTF8 | ENUM => StringType case null if assumeBinaryIsString => StringType case null => BinaryType case DECIMAL => makeDecimalType() diff --git a/sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetSchemaSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetSchemaSuite.scala index d0bfcde7e0..35d3c33f99 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetSchemaSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetSchemaSuite.scala @@ -161,6 +161,14 @@ class ParquetSchemaInferenceSuite extends ParquetSchemaTest { """.stripMargin, binaryAsString = true) + testSchemaInference[Tuple1[String]]( + "binary enum as string", + """ + |message root { + | optional binary _1 (ENUM); + |} + """.stripMargin) + testSchemaInference[Tuple1[Seq[Int]]]( "non-nullable array - non-standard", """ |