diff options
| author | Davies Liu <davies@databricks.com> | 2014-11-21 15:02:31 -0800 |
|---|---|---|
| committer | Xiangrui Meng <meng@databricks.com> | 2014-11-21 15:02:42 -0800 |
| commit | 9309ddfc3b9cca3780555fb3ac52d96343cb9545 (patch) | |
| tree | 9d2088c1347b94fe9d02331d2e4c389eaa300e92 /python/pyspark/mllib/regression.py | |
| parent | 6a01689a913a1a223fad66848c4fc17ab2931f22 (diff) | |
| download | spark-9309ddfc3b9cca3780555fb3ac52d96343cb9545.tar.gz spark-9309ddfc3b9cca3780555fb3ac52d96343cb9545.tar.bz2 spark-9309ddfc3b9cca3780555fb3ac52d96343cb9545.zip | |
[SPARK-4531] [MLlib] cache serialized java object
The Pyrolite is pretty slow (comparing to the adhoc serializer in 1.1), it cause much performance regression in 1.2, because we cache the serialized Python object in JVM, deserialize them into Java object in each step.
This PR change to cache the deserialized JavaRDD instead of PythonRDD to avoid the deserialization of Pyrolite. It should have similar memory usage as before, but much faster.
Author: Davies Liu <davies@databricks.com>
Closes #3397 from davies/cache and squashes the following commits:
7f6e6ce [Davies Liu] Update -> Updater
4b52edd [Davies Liu] using named argument
63b984e [Davies Liu] fix
7da0332 [Davies Liu] add unpersist()
dff33e1 [Davies Liu] address comments
c2bdfc2 [Davies Liu] refactor
d572f00 [Davies Liu] Merge branch 'master' into cache
f1063e1 [Davies Liu] cache serialized java object
(cherry picked from commit ce95bd8e130b2c7688b94be40683bdd90d86012d)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
Diffstat (limited to 'python/pyspark/mllib/regression.py')
| -rw-r--r-- | python/pyspark/mllib/regression.py | 5 |
1 files changed, 2 insertions, 3 deletions
diff --git a/python/pyspark/mllib/regression.py b/python/pyspark/mllib/regression.py index f4f5e615fa..210060140f 100644 --- a/python/pyspark/mllib/regression.py +++ b/python/pyspark/mllib/regression.py @@ -18,7 +18,7 @@ import numpy as np from numpy import array -from pyspark.mllib.common import callMLlibFunc, _to_java_object_rdd +from pyspark.mllib.common import callMLlibFunc from pyspark.mllib.linalg import SparseVector, _convert_to_vector __all__ = ['LabeledPoint', 'LinearModel', 'LinearRegressionModel', 'RidgeRegressionModel', @@ -129,8 +129,7 @@ def _regression_train_wrapper(train_func, modelClass, data, initial_weights): if not isinstance(first, LabeledPoint): raise ValueError("data should be an RDD of LabeledPoint, but got %s" % first) initial_weights = initial_weights or [0.0] * len(data.first().features) - weights, intercept = train_func(_to_java_object_rdd(data, cache=True), - _convert_to_vector(initial_weights)) + weights, intercept = train_func(data, _convert_to_vector(initial_weights)) return modelClass(weights, intercept) |