diff options
author | Tathagata Das <tathagata.das1565@gmail.com> | 2015-11-18 16:08:06 -0800 |
---|---|---|
committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-11-18 16:08:06 -0800 |
commit | a402c92c92b2e1c85d264f6077aec8f6d6a08270 (patch) | |
tree | 31f502bd6fed297f05de7ed60f5bde7de9663dd5 /make-distribution.sh | |
parent | 4b117121900e5f242e7c8f46a69164385f0da7cc (diff) | |
download | spark-a402c92c92b2e1c85d264f6077aec8f6d6a08270.tar.gz spark-a402c92c92b2e1c85d264f6077aec8f6d6a08270.tar.bz2 spark-a402c92c92b2e1c85d264f6077aec8f6d6a08270.zip |
[SPARK-11814][STREAMING] Add better default checkpoint duration
DStream checkpoint interval is by default set at max(10 second, batch interval). That's bad for large batch intervals where the checkpoint interval = batch interval, and RDDs get checkpointed every batch.
This PR is to set the checkpoint interval of trackStateByKey to 10 * batch duration.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #9805 from tdas/SPARK-11814.
Diffstat (limited to 'make-distribution.sh')
0 files changed, 0 insertions, 0 deletions