diff options
author | Wenchen Fan <wenchen@databricks.com> | 2017-03-28 11:47:43 -0700 |
---|---|---|
committer | Cheng Lian <lian@databricks.com> | 2017-03-28 11:47:43 -0700 |
commit | d4fac410e0554b7ccd44be44b7ce2fe07ed7f206 (patch) | |
tree | f7fdc2821ab2fb3a1aa3578f9951fa6773204eb6 /sql/core/src | |
parent | 17eddb35a280e77da7520343e0bf2a86b329ed62 (diff) | |
download | spark-d4fac410e0554b7ccd44be44b7ce2fe07ed7f206.tar.gz spark-d4fac410e0554b7ccd44be44b7ce2fe07ed7f206.tar.bz2 spark-d4fac410e0554b7ccd44be44b7ce2fe07ed7f206.zip |
[SPARK-20125][SQL] Dataset of type option of map does not work
## What changes were proposed in this pull request?
When we build the deserializer expression for map type, we will use `StaticInvoke` to call `ArrayBasedMapData.toScalaMap`, and declare the return type as `scala.collection.immutable.Map`. If the map is inside an Option, we will wrap this `StaticInvoke` with `WrapOption`, which requires the input to be `scala.collect.Map`. Ideally this should be fine, as `scala.collection.immutable.Map` extends `scala.collect.Map`, but our `ObjectType` is too strict about this, this PR fixes it.
## How was this patch tested?
new regression test
Author: Wenchen Fan <wenchen@databricks.com>
Closes #17454 from cloud-fan/map.
Diffstat (limited to 'sql/core/src')
-rw-r--r-- | sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala | 6 |
1 files changed, 6 insertions, 0 deletions
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala index 6417e7a8b6..68e071a1a6 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala @@ -1154,10 +1154,16 @@ class DatasetSuite extends QueryTest with SharedSQLContext { assert(errMsg3.getMessage.startsWith("cannot have circular references in class, but got the " + "circular reference of class")) } + + test("SPARK-20125: option of map") { + val ds = Seq(WithMapInOption(Some(Map(1 -> 1)))).toDS() + checkDataset(ds, WithMapInOption(Some(Map(1 -> 1)))) + } } case class WithImmutableMap(id: String, map_test: scala.collection.immutable.Map[Long, String]) case class WithMap(id: String, map_test: scala.collection.Map[Long, String]) +case class WithMapInOption(m: Option[scala.collection.Map[Int, Int]]) case class Generic[T](id: T, value: Double) |