diff options
| author | Michael Armbrust <michael@databricks.com> | 2014-11-20 20:34:43 -0800 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2014-11-20 20:34:55 -0800 |
| commit | 668643b8de0958094766fa62e7e2a7a0909f11da (patch) | |
| tree | 2588963f8981e4b064f9621b9b2eace5b4f87e9e /sql/catalyst | |
| parent | e445d3ce4e4fb9ee3c2feddb9734d541b61c6c01 (diff) | |
| download | spark-668643b8de0958094766fa62e7e2a7a0909f11da.tar.gz spark-668643b8de0958094766fa62e7e2a7a0909f11da.tar.bz2 spark-668643b8de0958094766fa62e7e2a7a0909f11da.zip | |
[SPARK-4522][SQL] Parse schema with missing metadata.
This is just a quick fix for 1.2. SPARK-4523 describes a more complete solution.
Author: Michael Armbrust <michael@databricks.com>
Closes #3392 from marmbrus/parquetMetadata and squashes the following commits:
bcc6626 [Michael Armbrust] Parse schema with missing metadata.
(cherry picked from commit 90a6a46bd11030672597f015dd443d954107123a)
Signed-off-by: Michael Armbrust <michael@databricks.com>
Diffstat (limited to 'sql/catalyst')
| -rw-r--r-- | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala | 6 |
1 files changed, 6 insertions, 0 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala index ff1dc03069..892b7e1a97 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala @@ -84,6 +84,12 @@ object DataType { ("nullable", JBool(nullable)), ("type", dataType: JValue)) => StructField(name, parseDataType(dataType), nullable, Metadata.fromJObject(metadata)) + // Support reading schema when 'metadata' is missing. + case JSortedObject( + ("name", JString(name)), + ("nullable", JBool(nullable)), + ("type", dataType: JValue)) => + StructField(name, parseDataType(dataType), nullable) } @deprecated("Use DataType.fromJson instead", "1.2.0") |