diff options
author | Tathagata Das <tathagata.das1565@gmail.com> | 2015-11-10 23:16:18 -0800 |
---|---|---|
committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-11-10 23:16:18 -0800 |
commit | 99f5f988612b3093d73d9ce98819767e822fcbff (patch) | |
tree | e69164304d215ade5b9c4e847e31ee295c61e62e /sql | |
parent | bd70244b3cda62cc447fd4cc343d4eb5ddaec893 (diff) | |
download | spark-99f5f988612b3093d73d9ce98819767e822fcbff.tar.gz spark-99f5f988612b3093d73d9ce98819767e822fcbff.tar.bz2 spark-99f5f988612b3093d73d9ce98819767e822fcbff.zip |
[SPARK-11290][STREAMING] Basic implementation of trackStateByKey
Current updateStateByKey provides stateful processing in Spark Streaming. It allows the user to maintain per-key state and manage that state using an updateFunction. The updateFunction is called for each key, and it uses new data and existing state of the key, to generate an updated state. However, based on community feedback, we have learnt the following lessons.
* Need for more optimized state management that does not scan every key
* Need to make it easier to implement common use cases - (a) timeout of idle data, (b) returning items other than state
The high level idea that of this PR
* Introduce a new API trackStateByKey that, allows the user to update per-key state, and emit arbitrary records. The new API is necessary as this will have significantly different semantics than the existing updateStateByKey API. This API will have direct support for timeouts.
* Internally, the system will keep the state data as a map/list within the partitions of the state RDDs. The new data RDDs will be partitioned appropriately, and for all the key-value data, it will lookup the map/list in the state RDD partition and create a new list/map of updated state data. The new state RDD partition will be created based on the update data and if necessary, with old data.
Here is the detailed design doc. Please take a look and provide feedback as comments.
https://docs.google.com/document/d/1NoALLyd83zGs1hNGMm0Pc5YOVgiPpMHugGMk6COqxxE/edit#heading=h.ph3w0clkd4em
This is still WIP. Major things left to be done.
- [x] Implement basic functionality of state tracking, with initial RDD and timeouts
- [x] Unit tests for state tracking
- [x] Unit tests for initial RDD and timeout
- [ ] Unit tests for TrackStateRDD
- [x] state creating, updating, removing
- [ ] emitting
- [ ] checkpointing
- [x] Misc unit tests for State, TrackStateSpec, etc.
- [x] Update docs and experimental tags
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #9256 from tdas/trackStateByKey.
Diffstat (limited to 'sql')
0 files changed, 0 insertions, 0 deletions