diff options
author | Sean Owen <sowen@cloudera.com> | 2016-02-19 10:26:38 +0000 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2016-02-19 10:26:38 +0000 |
commit | fb7e21797ed618d9754545a44f8f95f75b66757a (patch) | |
tree | 42b592cf1f25aeaf067c35afd75f9a3403182b99 /docs/streaming-programming-guide.md | |
parent | c776fce99b496a789ffcf2cfab78cf51eeea032b (diff) | |
download | spark-fb7e21797ed618d9754545a44f8f95f75b66757a.tar.gz spark-fb7e21797ed618d9754545a44f8f95f75b66757a.tar.bz2 spark-fb7e21797ed618d9754545a44f8f95f75b66757a.zip |
[SPARK-13339][DOCS] Clarify commutative / associative operator requirements for reduce, fold
Clarify that reduce functions need to be commutative, and fold functions do not
See https://github.com/apache/spark/pull/11091
Author: Sean Owen <sowen@cloudera.com>
Closes #11217 from srowen/SPARK-13339.
Diffstat (limited to 'docs/streaming-programming-guide.md')
-rw-r--r-- | docs/streaming-programming-guide.md | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/docs/streaming-programming-guide.md b/docs/streaming-programming-guide.md index 677f5ff7be..4d1932bc8c 100644 --- a/docs/streaming-programming-guide.md +++ b/docs/streaming-programming-guide.md @@ -798,7 +798,7 @@ Some of the common ones are as follows. <td> <b>reduce</b>(<i>func</i>) </td> <td> Return a new DStream of single-element RDDs by aggregating the elements in each RDD of the source DStream using a function <i>func</i> (which takes two arguments and returns one). - The function should be associative so that it can be computed in parallel. </td> + The function should be associative and commutative so that it can be computed in parallel. </td> </tr> <tr> <td> <b>countByValue</b>() </td> @@ -1072,7 +1072,7 @@ said two parameters - <i>windowLength</i> and <i>slideInterval</i>. <tr> <td> <b>reduceByWindow</b>(<i>func</i>, <i>windowLength</i>, <i>slideInterval</i>) </td> <td> Return a new single-element stream, created by aggregating elements in the stream over a - sliding interval using <i>func</i>. The function should be associative so that it can be computed + sliding interval using <i>func</i>. The function should be associative and commutative so that it can be computed correctly in parallel. </td> </tr> |