diff options
| author | Josh Rosen <joshrosen@eecs.berkeley.edu> | 2013-02-01 11:09:56 -0800 |
|---|---|---|
| committer | Josh Rosen <joshrosen@eecs.berkeley.edu> | 2013-02-01 11:13:10 -0800 |
| commit | 9cc6ff9c4e7eec2d62261fc166ad2ebade148752 (patch) | |
| tree | 2aea685655da9a2ed0acf4d7a40f81882e10b1e7 /python/pyspark/files.py | |
| parent | 571af31304bd72d310c3b47a8471a4de206aa6fe (diff) | |
| download | spark-9cc6ff9c4e7eec2d62261fc166ad2ebade148752.tar.gz spark-9cc6ff9c4e7eec2d62261fc166ad2ebade148752.tar.bz2 spark-9cc6ff9c4e7eec2d62261fc166ad2ebade148752.zip | |
Do not launch JavaGateways on workers (SPARK-674).
The problem was that the gateway was being initialized whenever the
pyspark.context module was loaded. The fix uses lazy initialization
that occurs only when SparkContext instances are actually constructed.
I also made the gateway and jvm variables private.
This change results in ~3-4x performance improvement when running the
PySpark unit tests.
Diffstat (limited to 'python/pyspark/files.py')
| -rw-r--r-- | python/pyspark/files.py | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/python/pyspark/files.py b/python/pyspark/files.py index 98f6a399cc..001b7a28b6 100644 --- a/python/pyspark/files.py +++ b/python/pyspark/files.py @@ -35,4 +35,4 @@ class SparkFiles(object): return cls._root_directory else: # This will have to change if we support multiple SparkContexts: - return cls._sc.jvm.spark.SparkFiles.getRootDirectory() + return cls._sc._jvm.spark.SparkFiles.getRootDirectory() |