aboutsummaryrefslogtreecommitdiff
path: root/docs/streaming-programming-guide.md
diff options
context:
space:
mode:
authorPatrick Wendell <pwendell@gmail.com>2013-02-25 15:09:46 -0800
committerPatrick Wendell <pwendell@gmail.com>2013-02-25 15:09:59 -0800
commit07f261876930a07161a7bcdb713430113e9e9ec8 (patch)
tree4f303955886b37afc395243c384c04eb4cfd996a /docs/streaming-programming-guide.md
parent271a40f0e435664be047e7c1506766b6789fcb78 (diff)
downloadspark-07f261876930a07161a7bcdb713430113e9e9ec8.tar.gz
spark-07f261876930a07161a7bcdb713430113e9e9ec8.tar.bz2
spark-07f261876930a07161a7bcdb713430113e9e9ec8.zip
Minor changes based on feedback
Diffstat (limited to 'docs/streaming-programming-guide.md')
-rw-r--r--docs/streaming-programming-guide.md4
1 files changed, 2 insertions, 2 deletions
diff --git a/docs/streaming-programming-guide.md b/docs/streaming-programming-guide.md
index d58d030ebd..4b351d07e5 100644
--- a/docs/streaming-programming-guide.md
+++ b/docs/streaming-programming-guide.md
@@ -286,9 +286,9 @@ For input streams that receive data from the network (that is, subclasses of Net
Note that, unlike RDDs, the default persistence level of DStreams keeps the data serialized in memory. This is further discussed in the [Performance Tuning](#memory-tuning) section. More information on different persistence levels can be found in [Spark Programming Guide](scala-programming-guide.html#rdd-persistence).
# RDD Checkpointing within DStreams
-A _stateful operation_ is one which operates over multiple batches of data. This includes all window-based operations and the `updateStateByKey` operation. Because stateful operations have an infinitely growing lineage, their state must be periodically checkpointed to HDFS.
+A _stateful operation_ is one which operates over multiple batches of data. This includes all window-based operations and the `updateStateByKey` operation.
-Checkpointing prevents extra computation during both failure recovery and normal operations by transparently truncating this lineage. Note that checkpointing also incurs the cost of saving to HDFS which may cause the corresponding batch to take longer to process. Hence, the interval of checkpointing needs to be set carefully. At small batch sizes (say 1 second), checkpointing every batch may significantly reduce operation throughput. Conversely, checkpointing too slowly causes the lineage and task sizes to grow which may have detrimental effects. Typically, a checkpoint interval of 5 - 10 times of sliding interval of a DStream is good setting to try.
+Because stateful operations have an infinitely growing lineage, their state must be periodically checkpointed to HDFS. Checkpointing truncates the lineage graph and stores the of the DStream in HDFS. Note that checkpointing also incurs the cost of saving to HDFS which may cause the corresponding batch to take longer to process. Hence, the interval of checkpointing needs to be set carefully. At small batch sizes (say 1 second), checkpointing every batch may significantly reduce operation throughput. Conversely, checkpointing too slowly causes the lineage and task sizes to grow which may have detrimental effects. Typically, a checkpoint interval of 5 - 10 times of sliding interval of a DStream is good setting to try.
To enable checkpointing, the developer has to provide the HDFS path to which RDD will be saved. This is done by using