diff options
| author | Josh Rosen <joshrosen@eecs.berkeley.edu> | 2013-02-01 11:09:56 -0800 |
|---|---|---|
| committer | Josh Rosen <joshrosen@eecs.berkeley.edu> | 2013-02-01 11:13:10 -0800 |
| commit | 9cc6ff9c4e7eec2d62261fc166ad2ebade148752 (patch) | |
| tree | 2aea685655da9a2ed0acf4d7a40f81882e10b1e7 /repl | |
| parent | 571af31304bd72d310c3b47a8471a4de206aa6fe (diff) | |
| download | spark-9cc6ff9c4e7eec2d62261fc166ad2ebade148752.tar.gz spark-9cc6ff9c4e7eec2d62261fc166ad2ebade148752.tar.bz2 spark-9cc6ff9c4e7eec2d62261fc166ad2ebade148752.zip | |
Do not launch JavaGateways on workers (SPARK-674).
The problem was that the gateway was being initialized whenever the
pyspark.context module was loaded. The fix uses lazy initialization
that occurs only when SparkContext instances are actually constructed.
I also made the gateway and jvm variables private.
This change results in ~3-4x performance improvement when running the
PySpark unit tests.
Diffstat (limited to 'repl')
0 files changed, 0 insertions, 0 deletions