diff options
author | Wenchen Fan <wenchen@databricks.com> | 2016-03-25 20:19:04 +0800 |
---|---|---|
committer | Cheng Lian <lian@databricks.com> | 2016-03-25 20:19:04 +0800 |
commit | e9b6e7d8577cd721a433130f29e8b112d98768b9 (patch) | |
tree | ebafcac42412b9411271d36088cf4e3c1313ada5 /repl/scala-2.11/src | |
parent | 70a6f0bb57ca2248444157e2707fbcc3cb04e3bc (diff) | |
download | spark-e9b6e7d8577cd721a433130f29e8b112d98768b9.tar.gz spark-e9b6e7d8577cd721a433130f29e8b112d98768b9.tar.bz2 spark-e9b6e7d8577cd721a433130f29e8b112d98768b9.zip |
[SPARK-13456][SQL][FOLLOW-UP] lazily generate the outer pointer for case class defined in REPL
## What changes were proposed in this pull request?
In https://github.com/apache/spark/pull/11410, we missed a corner case: define the inner class and use it in `Dataset` at the same time by using paste mode. For this case, the inner class and the `Dataset` are inside same line object, when we build the `Dataset`, we try to get outer pointer from line object, and it will fail because the line object is not initialized yet.
https://issues.apache.org/jira/browse/SPARK-13456?focusedCommentId=15209174&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15209174 is an example for this corner case.
This PR make the process of getting outer pointer from line object lazy, so that we can successfully build the `Dataset` and finish initializing the line object.
## How was this patch tested?
new test in repl suite.
Author: Wenchen Fan <wenchen@databricks.com>
Closes #11931 from cloud-fan/repl.
Diffstat (limited to 'repl/scala-2.11/src')
-rw-r--r-- | repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala | 15 |
1 files changed, 15 insertions, 0 deletions
diff --git a/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala b/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala index f148a6df47..dbfacba346 100644 --- a/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala +++ b/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala @@ -59,6 +59,10 @@ class ReplSuite extends SparkFunSuite { return out.toString } + // Simulate the paste mode in Scala REPL. + def runInterpreterInPasteMode(master: String, input: String): String = + runInterpreter(master, ":paste\n" + input + 4.toChar) // 4 is the ascii code of CTRL + D + def assertContains(message: String, output: String) { val isContain = output.contains(message) assert(isContain, @@ -381,4 +385,15 @@ class ReplSuite extends SparkFunSuite { assertDoesNotContain("error:", output) assertDoesNotContain("Exception", output) } + + test("define case class and create Dataset together with paste mode") { + val output = runInterpreterInPasteMode("local-cluster[1,1,1024]", + """ + |import sqlContext.implicits._ + |case class TestClass(value: Int) + |Seq(TestClass(1)).toDS() + """.stripMargin) + assertDoesNotContain("error:", output) + assertDoesNotContain("Exception", output) + } } |