diff options
author | Wenchen Fan <cloud0fan@outlook.com> | 2015-07-25 12:10:02 -0700 |
---|---|---|
committer | Reynold Xin <rxin@databricks.com> | 2015-07-25 12:10:02 -0700 |
commit | 2c94d0f24a37fa079b56d534b0b0a4574209215b (patch) | |
tree | 0aa5ca6ff2c9bb9728bfb6445bd4bfeb4cacf7b7 /core | |
parent | e2ec018e37cb699077b5fa2bd662f2055cb42296 (diff) | |
download | spark-2c94d0f24a37fa079b56d534b0b0a4574209215b.tar.gz spark-2c94d0f24a37fa079b56d534b0b0a4574209215b.tar.bz2 spark-2c94d0f24a37fa079b56d534b0b0a4574209215b.zip |
[SPARK-9192][SQL] add initialization phase for nondeterministic expression
Currently nondeterministic expression is broken without a explicit initialization phase.
Let me take `MonotonicallyIncreasingID` as an example. This expression need a mutable state to remember how many times it has been evaluated, so we use `transient var count: Long` there. By being transient, the `count` will be reset to 0 and **only** to 0 when serialize and deserialize it, as deserialize transient variable will result to default value. There is *no way* to use another initial value for `count`, until we add the explicit initialization phase.
Another use case is local execution for `LocalRelation`, there is no serialize and deserialize phase and thus we can't reset mutable states for it.
Author: Wenchen Fan <cloud0fan@outlook.com>
Closes #7535 from cloud-fan/init and squashes the following commits:
6c6f332 [Wenchen Fan] add test
ef68ff4 [Wenchen Fan] fix comments
9eac85e [Wenchen Fan] move init code to interpreted class
bb7d838 [Wenchen Fan] pulls out nondeterministic expressions into a project
b4a4fc7 [Wenchen Fan] revert a refactor
86fee36 [Wenchen Fan] add initialization phase for nondeterministic expression
Diffstat (limited to 'core')
0 files changed, 0 insertions, 0 deletions