From 119b06a04f6df3949b3b074a18f791bbc732ac31 Mon Sep 17 00:00:00 2001 From: Ali Ghodsi Date: Sun, 15 Jun 2014 23:44:30 -0700 Subject: Updating docs to include missing information about reducers and clarify ... ...how the OFFHEAP storage level works (there has been confusion around this). Author: Ali Ghodsi Closes #1089 from alig/master and squashes the following commits: ca8114d [Ali Ghodsi] Updating docs to include missing information about reducers and clarify how the OFFHEAP storage level works (there has been confusion around this). --- docs/programming-guide.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) (limited to 'docs/programming-guide.md') diff --git a/docs/programming-guide.md b/docs/programming-guide.md index 0b24a8b88b..65d75b85ef 100644 --- a/docs/programming-guide.md +++ b/docs/programming-guide.md @@ -899,7 +899,7 @@ for details. reduceByKey(func, [numTasks]) - When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function. Like in groupByKey, the number of reduce tasks is configurable through an optional second argument. + When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function func, which must be of type (V,V) => V. Like in groupByKey, the number of reduce tasks is configurable through an optional second argument. aggregateByKey(zeroValue)(seqOp, combOp, [numTasks]) @@ -1067,7 +1067,10 @@ storage levels is: Store RDD in serialized format in Tachyon. Compared to MEMORY_ONLY_SER, OFF_HEAP reduces garbage collection overhead and allows executors to be smaller and to share a pool of memory, making it attractive in environments with - large heaps or multiple concurrent applications. + large heaps or multiple concurrent applications. Furthermore, as the RDDs reside in Tachyon, + the crash of an executor does not lead to losing the in-memory cache. In this mode, the memory + in Tachyon is discardable. Thus, Tachyon does not attempt to reconstruct a block that it evicts + from memory. -- cgit v1.2.3