aboutsummaryrefslogtreecommitdiff
path: root/repl
diff options
context:
space:
mode:
authorWenchen Fan <wenchen@databricks.com>2016-03-25 20:19:04 +0800
committerCheng Lian <lian@databricks.com>2016-03-25 20:19:04 +0800
commite9b6e7d8577cd721a433130f29e8b112d98768b9 (patch)
treeebafcac42412b9411271d36088cf4e3c1313ada5 /repl
parent70a6f0bb57ca2248444157e2707fbcc3cb04e3bc (diff)
downloadspark-e9b6e7d8577cd721a433130f29e8b112d98768b9.tar.gz
spark-e9b6e7d8577cd721a433130f29e8b112d98768b9.tar.bz2
spark-e9b6e7d8577cd721a433130f29e8b112d98768b9.zip
[SPARK-13456][SQL][FOLLOW-UP] lazily generate the outer pointer for case class defined in REPL
## What changes were proposed in this pull request? In https://github.com/apache/spark/pull/11410, we missed a corner case: define the inner class and use it in `Dataset` at the same time by using paste mode. For this case, the inner class and the `Dataset` are inside same line object, when we build the `Dataset`, we try to get outer pointer from line object, and it will fail because the line object is not initialized yet. https://issues.apache.org/jira/browse/SPARK-13456?focusedCommentId=15209174&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15209174 is an example for this corner case. This PR make the process of getting outer pointer from line object lazy, so that we can successfully build the `Dataset` and finish initializing the line object. ## How was this patch tested? new test in repl suite. Author: Wenchen Fan <wenchen@databricks.com> Closes #11931 from cloud-fan/repl.
Diffstat (limited to 'repl')
-rw-r--r--repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala15
1 files changed, 15 insertions, 0 deletions
diff --git a/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala b/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
index f148a6df47..dbfacba346 100644
--- a/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
+++ b/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
@@ -59,6 +59,10 @@ class ReplSuite extends SparkFunSuite {
return out.toString
}
+ // Simulate the paste mode in Scala REPL.
+ def runInterpreterInPasteMode(master: String, input: String): String =
+ runInterpreter(master, ":paste\n" + input + 4.toChar) // 4 is the ascii code of CTRL + D
+
def assertContains(message: String, output: String) {
val isContain = output.contains(message)
assert(isContain,
@@ -381,4 +385,15 @@ class ReplSuite extends SparkFunSuite {
assertDoesNotContain("error:", output)
assertDoesNotContain("Exception", output)
}
+
+ test("define case class and create Dataset together with paste mode") {
+ val output = runInterpreterInPasteMode("local-cluster[1,1,1024]",
+ """
+ |import sqlContext.implicits._
+ |case class TestClass(value: Int)
+ |Seq(TestClass(1)).toDS()
+ """.stripMargin)
+ assertDoesNotContain("error:", output)
+ assertDoesNotContain("Exception", output)
+ }
}