diff options
| author | Davies Liu <davies@databricks.com> | 2015-04-15 12:58:02 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-04-15 12:58:02 -0700 |
| commit | f11288d5272bc18585b8cad4ee3bd59eade7c296 (patch) | |
| tree | 0016cd5ccb7c7025d00ce79c3aefc3b9799420c5 /sql | |
| parent | 6c5ed8a6d552abd967d27cdb94b68d46ccb57221 (diff) | |
| download | spark-f11288d5272bc18585b8cad4ee3bd59eade7c296.tar.gz spark-f11288d5272bc18585b8cad4ee3bd59eade7c296.tar.bz2 spark-f11288d5272bc18585b8cad4ee3bd59eade7c296.zip | |
[SPARK-6886] [PySpark] fix big closure with shuffle
Currently, the created broadcast object will have same life cycle as RDD in Python. For multistage jobs, an PythonRDD will be created in JVM and the RDD in Python may be GCed, then the broadcast will be destroyed in JVM before the PythonRDD.
This PR change to use PythonRDD to track the lifecycle of the broadcast object. It also have a refactor about getNumPartitions() to avoid unnecessary creation of PythonRDD, which could be heavy.
cc JoshRosen
Author: Davies Liu <davies@databricks.com>
Closes #5496 from davies/big_closure and squashes the following commits:
9a0ea4c [Davies Liu] fix big closure with shuffle
Diffstat (limited to 'sql')
0 files changed, 0 insertions, 0 deletions