diff options
| author | Dongjoon Hyun <dongjoon@apache.org> | 2016-03-07 12:06:46 -0800 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2016-03-07 12:06:46 -0800 |
| commit | e72914f37de85519fc2aa131bac69d7582de98c8 (patch) | |
| tree | 2368b61ef20841707bf22047ef094b942e729b6c /docs | |
| parent | ef77003178eb5cdcb4fe519fc540917656c5d577 (diff) | |
| download | spark-e72914f37de85519fc2aa131bac69d7582de98c8.tar.gz spark-e72914f37de85519fc2aa131bac69d7582de98c8.tar.bz2 spark-e72914f37de85519fc2aa131bac69d7582de98c8.zip | |
[SPARK-12243][BUILD][PYTHON] PySpark tests are slow in Jenkins.
## What changes were proposed in this pull request?
In the Jenkins pull request builder, PySpark tests take around [962 seconds ](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52530/console) of end-to-end time to run, despite the fact that we run four Python test suites in parallel. According to the log, the basic reason is that the long running test starts at the end due to FIFO queue. We first try to reduce the test time by just starting some long running tests first with simple priority queue.
```
========================================================================
Running PySpark tests
========================================================================
...
Finished test(python3.4): pyspark.streaming.tests (213s)
Finished test(pypy): pyspark.sql.tests (92s)
Finished test(pypy): pyspark.streaming.tests (280s)
Tests passed in 962 seconds
```
## How was this patch tested?
Manual check.
Check 'Running PySpark tests' part of the Jenkins log.
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #11551 from dongjoon-hyun/SPARK-12243.
Diffstat (limited to 'docs')
0 files changed, 0 insertions, 0 deletions