diff options
author | Davies Liu <davies.liu@gmail.com> | 2014-10-18 19:14:48 -0700 |
---|---|---|
committer | Josh Rosen <joshrosen@databricks.com> | 2014-10-18 19:14:48 -0700 |
commit | 05db2da7dc256822cdb602c4821cbb9fb84dac98 (patch) | |
tree | d80715dff7672792539f9aaa66858999100324f5 /python | |
parent | f406a8391825d8866110f29a0d656c82cd064520 (diff) | |
download | spark-05db2da7dc256822cdb602c4821cbb9fb84dac98.tar.gz spark-05db2da7dc256822cdb602c4821cbb9fb84dac98.tar.bz2 spark-05db2da7dc256822cdb602c4821cbb9fb84dac98.zip |
[SPARK-3952] [Streaming] [PySpark] add Python examples in Streaming Programming Guide
Having Python examples in Streaming Programming Guide.
Also add RecoverableNetworkWordCount example.
Author: Davies Liu <davies.liu@gmail.com>
Author: Davies Liu <davies@databricks.com>
Closes #2808 from davies/pyguide and squashes the following commits:
8d4bec4 [Davies Liu] update readme
26a7e37 [Davies Liu] fix format
3821c4d [Davies Liu] address comments, add missing file
7e4bb8a [Davies Liu] add Python examples in Streaming Programming Guide
Diffstat (limited to 'python')
-rw-r--r-- | python/docs/pyspark.streaming.rst | 10 | ||||
-rw-r--r-- | python/pyspark/streaming/dstream.py | 8 |
2 files changed, 15 insertions, 3 deletions
diff --git a/python/docs/pyspark.streaming.rst b/python/docs/pyspark.streaming.rst new file mode 100644 index 0000000000..5024d694b6 --- /dev/null +++ b/python/docs/pyspark.streaming.rst @@ -0,0 +1,10 @@ +pyspark.streaming module +================== + +Module contents +--------------- + +.. automodule:: pyspark.streaming + :members: + :undoc-members: + :show-inheritance: diff --git a/python/pyspark/streaming/dstream.py b/python/pyspark/streaming/dstream.py index 5ae5cf07f0..0826ddc56e 100644 --- a/python/pyspark/streaming/dstream.py +++ b/python/pyspark/streaming/dstream.py @@ -441,9 +441,11 @@ class DStream(object): if `invReduceFunc` is not None, the reduction is done incrementally using the old window's reduced value : - 1. reduce the new values that entered the window (e.g., adding new counts) - 2. "inverse reduce" the old values that left the window (e.g., subtracting old counts) - This is more efficient than `invReduceFunc` is None. + + 1. reduce the new values that entered the window (e.g., adding new counts) + + 2. "inverse reduce" the old values that left the window (e.g., subtracting old counts) + This is more efficient than `invReduceFunc` is None. @param reduceFunc: associative reduce function @param invReduceFunc: inverse reduce function of `reduceFunc` |