diff options
| author | Davies Liu <davies@databricks.com> | 2015-01-29 17:28:37 -0800 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-01-29 17:28:37 -0800 |
| commit | 5c746eedda8cff2fc1692cf6dce376f4b0ca6fac (patch) | |
| tree | d4474477ae270cb6059da20a4706b5c45e3ec787 /sql | |
| parent | ce9c43ba8ca1ba6507fd3bf3c647ab7396d33653 (diff) | |
| download | spark-5c746eedda8cff2fc1692cf6dce376f4b0ca6fac.tar.gz spark-5c746eedda8cff2fc1692cf6dce376f4b0ca6fac.tar.bz2 spark-5c746eedda8cff2fc1692cf6dce376f4b0ca6fac.zip | |
[SPARK-5395] [PySpark] fix python process leak while coalesce()
Currently, the Python process is released into pool only after the task had finished, it cause many process forked if coalesce() is called.
This PR will change it to release the process as soon as read all the data from it (finish the partition), then a process could be reused to process multiple partitions in a single task.
Author: Davies Liu <davies@databricks.com>
Closes #4238 from davies/py_leak and squashes the following commits:
ec80a43 [Davies Liu] add @volatile
6da437a [Davies Liu] address comments
24ed322 [Davies Liu] fix python process leak while coalesce()
Diffstat (limited to 'sql')
0 files changed, 0 insertions, 0 deletions