diff options
| author | Xin Ren <iamshrek@126.com> | 2016-05-10 15:12:47 -0700 |
|---|---|---|
| committer | Shixiong Zhu <shixiong@databricks.com> | 2016-05-10 15:12:47 -0700 |
| commit | 86475520f88f90c9d3b71516f65ccc0e9d244863 (patch) | |
| tree | 365a7f00639c95b46c330ab30215c24f41ac1796 /python | |
| parent | da02d006bbb5c4fe62abd5542b9fff7d1c58603c (diff) | |
| download | spark-86475520f88f90c9d3b71516f65ccc0e9d244863.tar.gz spark-86475520f88f90c9d3b71516f65ccc0e9d244863.tar.bz2 spark-86475520f88f90c9d3b71516f65ccc0e9d244863.zip | |
[SPARK-14936][BUILD][TESTS] FlumePollingStreamSuite is slow
https://issues.apache.org/jira/browse/SPARK-14936
## What changes were proposed in this pull request?
FlumePollingStreamSuite contains two tests which run for a minute each. This seems excessively slow and we should speed it up if possible.
In this PR, instead of creating `StreamingContext` directly from `conf`, here an underlying `SparkContext` is created before all and it is used to create each`StreamingContext`.
Running time is reduced by avoiding multiple `SparkContext` creations and destroys.
## How was this patch tested?
Tested on my local machine running `testOnly *.FlumePollingStreamSuite`
Author: Xin Ren <iamshrek@126.com>
Closes #12845 from keypointt/SPARK-14936.
Diffstat (limited to 'python')
| -rw-r--r-- | python/pyspark/streaming/tests.py | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/python/pyspark/streaming/tests.py b/python/pyspark/streaming/tests.py index 148bf7e8ff..f27628c895 100644 --- a/python/pyspark/streaming/tests.py +++ b/python/pyspark/streaming/tests.py @@ -1357,7 +1357,7 @@ class FlumePollingStreamTests(PySparkStreamingTestCase): dstream.foreachRDD(get_output) ssc.start() - self._utils.sendDatAndEnsureAllDataHasBeenReceived() + self._utils.sendDataAndEnsureAllDataHasBeenReceived() self.wait_for(outputBuffer, self._utils.getTotalEvents()) outputHeaders = [event[0] for event in outputBuffer] |