diff options
| author | Reynold Xin <rxin@databricks.com> | 2015-09-15 16:53:27 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-09-15 16:53:27 -0700 |
| commit | a63cdc769f511e98b38c3318bcc732c9a6c76c22 (patch) | |
| tree | 9fe90e3343d8c2fe332961928006cab59973c9a3 /sql | |
| parent | b6e998634e05db0bb6267173e7b28f885c808c16 (diff) | |
| download | spark-a63cdc769f511e98b38c3318bcc732c9a6c76c22.tar.gz spark-a63cdc769f511e98b38c3318bcc732c9a6c76c22.tar.bz2 spark-a63cdc769f511e98b38c3318bcc732c9a6c76c22.zip | |
[SPARK-10612] [SQL] Add prepare to LocalNode.
The idea is that we should separate the function call that does memory reservation (i.e. prepare) from the function call that consumes the input (e.g. open()), so all operators can be a chance to reserve memory before they are all consumed.
Author: Reynold Xin <rxin@databricks.com>
Closes #8761 from rxin/SPARK-10612.
Diffstat (limited to 'sql')
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/execution/local/LocalNode.scala | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/local/LocalNode.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/local/LocalNode.scala index 9840080e16..569cff565c 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/local/LocalNode.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/local/LocalNode.scala @@ -46,6 +46,14 @@ abstract class LocalNode(conf: SQLConf) extends TreeNode[LocalNode] with Logging def output: Seq[Attribute] /** + * Called before open(). Prepare can be used to reserve memory needed. It must NOT consume + * any input data. + * + * Implementations of this must also call the `prepare()` function of its children. + */ + def prepare(): Unit = children.foreach(_.prepare()) + + /** * Initializes the iterator state. Must be called before calling `next()`. * * Implementations of this must also call the `open()` function of its children. |