diff options
| author | Tathagata Das <tathagata.das1565@gmail.com> | 2015-08-14 15:10:01 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-08-14 15:10:01 -0700 |
| commit | f3bfb711c1742d0915e43bda8230b4d1d22b4190 (patch) | |
| tree | c6339008ed647f134b8fcca88e78afab81e865c0 /.gitattributes | |
| parent | 1150a19b188a075166899fdb1e107b2ba1e505d8 (diff) | |
| download | spark-f3bfb711c1742d0915e43bda8230b4d1d22b4190.tar.gz spark-f3bfb711c1742d0915e43bda8230b4d1d22b4190.tar.bz2 spark-f3bfb711c1742d0915e43bda8230b4d1d22b4190.zip | |
[SPARK-9966] [STREAMING] Handle couple of corner cases in PIDRateEstimator
1. The rate estimator should not estimate any rate when there are no records in the batch, as there is no data to estimate the rate. In the current state, it estimates and set the rate to zero. That is incorrect.
2. The rate estimator should not never set the rate to zero under any circumstances. Otherwise the system will stop receiving data, and stop generating useful estimates (see reason 1). So the fix is to define a parameters that sets a lower bound on the estimated rate, so that the system always receives some data.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #8199 from tdas/SPARK-9966 and squashes the following commits:
829f793 [Tathagata Das] Fixed unit test and added comments
3a994db [Tathagata Das] Added min rate and updated tests in PIDRateEstimator
Diffstat (limited to '.gitattributes')
0 files changed, 0 insertions, 0 deletions