diff options
| author | junhao <junhao@mogujie.com> | 2016-02-16 19:43:17 -0800 |
|---|---|---|
| committer | Shixiong Zhu <shixiong@databricks.com> | 2016-02-16 19:43:17 -0800 |
| commit | 7218c0eba957e0a079a407b79c3a050cce9647b2 (patch) | |
| tree | fa8fdf3612c4c3d0ed36dcf131931a48d8108ee1 /core | |
| parent | 5f37aad48cb729a80c4cc25347460f12aafec9fb (diff) | |
| download | spark-7218c0eba957e0a079a407b79c3a050cce9647b2.tar.gz spark-7218c0eba957e0a079a407b79c3a050cce9647b2.tar.bz2 spark-7218c0eba957e0a079a407b79c3a050cce9647b2.zip | |
[SPARK-11627] Add initial input rate limit for spark streaming backpressure mechanism.
https://issues.apache.org/jira/browse/SPARK-11627
Spark Streaming backpressure mechanism has no initial input rate limit, it might cause OOM exception.
In the firest batch task ,receivers receive data at the maximum speed they can reach,it might exhaust executors memory resources. Add a initial input rate limit value can make sure the Streaming job execute success in the first batch,then the backpressure mechanism can adjust receiving rate adaptively.
Author: junhao <junhao@mogujie.com>
Closes #9593 from junhaoMg/junhao-dev.
Diffstat (limited to 'core')
0 files changed, 0 insertions, 0 deletions