diff options
| author | Michael Armbrust <michael@databricks.com> | 2014-11-20 20:34:43 -0800 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2014-11-20 20:34:43 -0800 |
| commit | 90a6a46bd11030672597f015dd443d954107123a (patch) | |
| tree | 4fe365d74497c096d4adc999fe6a10ab5e90c33e /sql/catalyst | |
| parent | 8cd6eea6298fc8e811dece38c2875e94ff863948 (diff) | |
| download | spark-90a6a46bd11030672597f015dd443d954107123a.tar.gz spark-90a6a46bd11030672597f015dd443d954107123a.tar.bz2 spark-90a6a46bd11030672597f015dd443d954107123a.zip | |
[SPARK-4522][SQL] Parse schema with missing metadata.
This is just a quick fix for 1.2. SPARK-4523 describes a more complete solution.
Author: Michael Armbrust <michael@databricks.com>
Closes #3392 from marmbrus/parquetMetadata and squashes the following commits:
bcc6626 [Michael Armbrust] Parse schema with missing metadata.
Diffstat (limited to 'sql/catalyst')
| -rw-r--r-- | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala | 6 |
1 files changed, 6 insertions, 0 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala index ff1dc03069..892b7e1a97 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala @@ -84,6 +84,12 @@ object DataType { ("nullable", JBool(nullable)), ("type", dataType: JValue)) => StructField(name, parseDataType(dataType), nullable, Metadata.fromJObject(metadata)) + // Support reading schema when 'metadata' is missing. + case JSortedObject( + ("name", JString(name)), + ("nullable", JBool(nullable)), + ("type", dataType: JValue)) => + StructField(name, parseDataType(dataType), nullable) } @deprecated("Use DataType.fromJson instead", "1.2.0") |