diff options
author | Yin Huai <yhuai@databricks.com> | 2015-12-01 17:18:45 -0800 |
---|---|---|
committer | Michael Armbrust <michael@databricks.com> | 2015-12-01 17:18:45 -0800 |
commit | e96a70d5ab2e2b43a2df17a550fa9ed2ee0001c4 (patch) | |
tree | b326c866a2dc1e78b3fa674ec87e71c4734b2db5 /core | |
parent | 5872a9d89fe2720c2bcb1fc7494136947a72581c (diff) | |
download | spark-e96a70d5ab2e2b43a2df17a550fa9ed2ee0001c4.tar.gz spark-e96a70d5ab2e2b43a2df17a550fa9ed2ee0001c4.tar.bz2 spark-e96a70d5ab2e2b43a2df17a550fa9ed2ee0001c4.zip |
[SPARK-11596][SQL] In TreeNode's argString, if a TreeNode is not a child of the current TreeNode, we should only return the simpleString.
In TreeNode's argString, if a TreeNode is not a child of the current TreeNode, we will only return the simpleString.
I tested the [following case provided by Cristian](https://issues.apache.org/jira/browse/SPARK-11596?focusedCommentId=15019241&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15019241).
```
val c = (1 to 20).foldLeft[Option[DataFrame]] (None) { (curr, idx) =>
println(s"PROCESSING >>>>>>>>>>> $idx")
val df = sqlContext.sparkContext.parallelize((0 to 10).zipWithIndex).toDF("A", "B")
val union = curr.map(_.unionAll(df)).getOrElse(df)
union.cache()
Some(union)
}
c.get.explain(true)
```
Without the change, `c.get.explain(true)` took 100s. With the change, `c.get.explain(true)` took 26ms.
https://issues.apache.org/jira/browse/SPARK-11596
Author: Yin Huai <yhuai@databricks.com>
Closes #10079 from yhuai/SPARK-11596.
Diffstat (limited to 'core')
0 files changed, 0 insertions, 0 deletions