aboutsummaryrefslogtreecommitdiff
path: root/docs/streaming-programming-guide.md
diff options
context:
space:
mode:
authorPatrick Wendell <pwendell@gmail.com>2013-02-25 15:24:17 -0800
committerPatrick Wendell <pwendell@gmail.com>2013-02-25 15:24:17 -0800
commit918ee258673b2364730fe38fe781cb5167870e7a (patch)
tree663884e5ca4f574ea38cf4da8748f9716ace910b /docs/streaming-programming-guide.md
parent07f261876930a07161a7bcdb713430113e9e9ec8 (diff)
downloadspark-918ee258673b2364730fe38fe781cb5167870e7a.tar.gz
spark-918ee258673b2364730fe38fe781cb5167870e7a.tar.bz2
spark-918ee258673b2364730fe38fe781cb5167870e7a.zip
One more change done with TD
Diffstat (limited to 'docs/streaming-programming-guide.md')
-rw-r--r--docs/streaming-programming-guide.md2
1 files changed, 1 insertions, 1 deletions
diff --git a/docs/streaming-programming-guide.md b/docs/streaming-programming-guide.md
index 4b351d07e5..e1407a49b0 100644
--- a/docs/streaming-programming-guide.md
+++ b/docs/streaming-programming-guide.md
@@ -288,7 +288,7 @@ Note that, unlike RDDs, the default persistence level of DStreams keeps the data
# RDD Checkpointing within DStreams
A _stateful operation_ is one which operates over multiple batches of data. This includes all window-based operations and the `updateStateByKey` operation.
-Because stateful operations have an infinitely growing lineage, their state must be periodically checkpointed to HDFS. Checkpointing truncates the lineage graph and stores the of the DStream in HDFS. Note that checkpointing also incurs the cost of saving to HDFS which may cause the corresponding batch to take longer to process. Hence, the interval of checkpointing needs to be set carefully. At small batch sizes (say 1 second), checkpointing every batch may significantly reduce operation throughput. Conversely, checkpointing too slowly causes the lineage and task sizes to grow which may have detrimental effects. Typically, a checkpoint interval of 5 - 10 times of sliding interval of a DStream is good setting to try.
+Because stateful operations have a dependency on previous batches of data, they continuously accumulate metadata over time. To clear this meta-data, streaming supports periodic _checkpointing_ by saving intermediate data to HDFS. Note that checkpointing also incurs the cost of saving to HDFS which may cause the corresponding batch to take longer to process. Hence, the interval of checkpointing needs to be set carefully. At small batch sizes (say 1 second), checkpointing every batch may significantly reduce operation throughput. Conversely, checkpointing too slowly causes the lineage and task sizes to grow which may have detrimental effects. Typically, a checkpoint interval of 5 - 10 times of sliding interval of a DStream is good setting to try.
To enable checkpointing, the developer has to provide the HDFS path to which RDD will be saved. This is done by using