From fb7e21797ed618d9754545a44f8f95f75b66757a Mon Sep 17 00:00:00 2001 From: Sean Owen Date: Fri, 19 Feb 2016 10:26:38 +0000 Subject: [SPARK-13339][DOCS] Clarify commutative / associative operator requirements for reduce, fold Clarify that reduce functions need to be commutative, and fold functions do not See https://github.com/apache/spark/pull/11091 Author: Sean Owen Closes #11217 from srowen/SPARK-13339. --- docs/programming-guide.md | 2 +- docs/streaming-programming-guide.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) (limited to 'docs') diff --git a/docs/programming-guide.md b/docs/programming-guide.md index e45081464a..2d6f7767d9 100644 --- a/docs/programming-guide.md +++ b/docs/programming-guide.md @@ -1343,7 +1343,7 @@ value of the broadcast variable (e.g. if the variable is shipped to a new node l ## Accumulators -Accumulators are variables that are only "added" to through an associative operation and can +Accumulators are variables that are only "added" to through an associative and commutative operation and can therefore be efficiently supported in parallel. They can be used to implement counters (as in MapReduce) or sums. Spark natively supports accumulators of numeric types, and programmers can add support for new types. If accumulators are created with a name, they will be diff --git a/docs/streaming-programming-guide.md b/docs/streaming-programming-guide.md index 677f5ff7be..4d1932bc8c 100644 --- a/docs/streaming-programming-guide.md +++ b/docs/streaming-programming-guide.md @@ -798,7 +798,7 @@ Some of the common ones are as follows. reduce(func) Return a new DStream of single-element RDDs by aggregating the elements in each RDD of the source DStream using a function func (which takes two arguments and returns one). - The function should be associative so that it can be computed in parallel. + The function should be associative and commutative so that it can be computed in parallel. countByValue() @@ -1072,7 +1072,7 @@ said two parameters - windowLength and slideInterval. reduceByWindow(func, windowLength, slideInterval) Return a new single-element stream, created by aggregating elements in the stream over a - sliding interval using func. The function should be associative so that it can be computed + sliding interval using func. The function should be associative and commutative so that it can be computed correctly in parallel. -- cgit v1.2.3