diff options
| author | Josh Rosen <joshrosen@eecs.berkeley.edu> | 2013-02-01 11:09:56 -0800 |
|---|---|---|
| committer | Josh Rosen <joshrosen@eecs.berkeley.edu> | 2013-02-01 11:13:10 -0800 |
| commit | 9cc6ff9c4e7eec2d62261fc166ad2ebade148752 (patch) | |
| tree | 2aea685655da9a2ed0acf4d7a40f81882e10b1e7 /python/pyspark/tests.py | |
| parent | 571af31304bd72d310c3b47a8471a4de206aa6fe (diff) | |
| download | spark-9cc6ff9c4e7eec2d62261fc166ad2ebade148752.tar.gz spark-9cc6ff9c4e7eec2d62261fc166ad2ebade148752.tar.bz2 spark-9cc6ff9c4e7eec2d62261fc166ad2ebade148752.zip | |
Do not launch JavaGateways on workers (SPARK-674).
The problem was that the gateway was being initialized whenever the
pyspark.context module was loaded. The fix uses lazy initialization
that occurs only when SparkContext instances are actually constructed.
I also made the gateway and jvm variables private.
This change results in ~3-4x performance improvement when running the
PySpark unit tests.
Diffstat (limited to 'python/pyspark/tests.py')
| -rw-r--r-- | python/pyspark/tests.py | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/python/pyspark/tests.py b/python/pyspark/tests.py index 52297d44e6..6a1962d267 100644 --- a/python/pyspark/tests.py +++ b/python/pyspark/tests.py @@ -26,7 +26,7 @@ class PySparkTestCase(unittest.TestCase): sys.path = self._old_sys_path # To avoid Akka rebinding to the same port, since it doesn't unbind # immediately on shutdown - self.sc.jvm.System.clearProperty("spark.driver.port") + self.sc._jvm.System.clearProperty("spark.driver.port") class TestCheckpoint(PySparkTestCase): |