diff options
| author | Tathagata Das <tathagata.das1565@gmail.com> | 2016-10-21 13:07:29 -0700 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2016-10-21 13:07:29 -0700 |
| commit | 7a531e3054f8d4820216ed379433559f57f571b8 (patch) | |
| tree | 8113aa776cc65b3f18dbb5ce3bb63eb3e61aa1f5 /python/pyspark | |
| parent | e371040a0150e4ed748a7c25465965840b61ca63 (diff) | |
| download | spark-7a531e3054f8d4820216ed379433559f57f571b8.tar.gz spark-7a531e3054f8d4820216ed379433559f57f571b8.tar.bz2 spark-7a531e3054f8d4820216ed379433559f57f571b8.zip | |
[SPARK-17926][SQL][STREAMING] Added json for statuses
## What changes were proposed in this pull request?
StreamingQueryStatus exposed through StreamingQueryListener often needs to be recorded (similar to SparkListener events). This PR adds `.json` and `.prettyJson` to `StreamingQueryStatus`, `SourceStatus` and `SinkStatus`.
## How was this patch tested?
New unit tests
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #15476 from tdas/SPARK-17926.
Diffstat (limited to 'python/pyspark')
| -rw-r--r-- | python/pyspark/sql/streaming.py | 11 |
1 files changed, 5 insertions, 6 deletions
diff --git a/python/pyspark/sql/streaming.py b/python/pyspark/sql/streaming.py index ce47bd1640..35fc469291 100644 --- a/python/pyspark/sql/streaming.py +++ b/python/pyspark/sql/streaming.py @@ -205,8 +205,7 @@ class StreamingQueryStatus(object): Pretty string of this query status. >>> print(sqs) - StreamingQueryStatus: - Query name: query + Status of query 'query' Query id: 1 Status timestamp: 123 Input rate: 15.5 rows/sec @@ -220,7 +219,7 @@ class StreamingQueryStatus(object): numRows.input.total: 100 triggerId: 5 Source statuses [1 source]: - Source 1: MySource1 + Source 1 - MySource1 Available offset: #0 Input rate: 15.5 rows/sec Processing rate: 23.5 rows/sec @@ -228,7 +227,7 @@ class StreamingQueryStatus(object): numRows.input.source: 100 latency.getOffset.source: 10 latency.getBatch.source: 20 - Sink status: MySink + Sink status - MySink Committed offsets: [#1, -] """ return self._jsqs.toString() @@ -366,7 +365,7 @@ class SourceStatus(object): Pretty string of this source status. >>> print(sqs.sourceStatuses[0]) - SourceStatus: MySource1 + Status of source MySource1 Available offset: #0 Input rate: 15.5 rows/sec Processing rate: 23.5 rows/sec @@ -457,7 +456,7 @@ class SinkStatus(object): Pretty string of this source status. >>> print(sqs.sinkStatus) - SinkStatus: MySink + Status of sink MySink Committed offsets: [#1, -] """ return self._jss.toString() |