diff options
| author | Xiangrui Meng <meng@databricks.com> | 2015-05-28 22:38:38 -0700 |
|---|---|---|
| committer | Xiangrui Meng <meng@databricks.com> | 2015-05-28 22:38:38 -0700 |
| commit | db9513789756da4f16bb1fe8cf1d19500f231f54 (patch) | |
| tree | aaef83386cdad3975181b554d68527abf41407cb /python/pyspark/mllib/common.py | |
| parent | cd3d9a5c0c3e77098a72c85dffe4a27737009ae7 (diff) | |
| download | spark-db9513789756da4f16bb1fe8cf1d19500f231f54.tar.gz spark-db9513789756da4f16bb1fe8cf1d19500f231f54.tar.bz2 spark-db9513789756da4f16bb1fe8cf1d19500f231f54.zip | |
[SPARK-7922] [MLLIB] use DataFrames for user/item factors in ALSModel
Expose user/item factors in DataFrames. This is to be more consistent with the pipeline API. It also helps maintain consistent APIs across languages. This PR also removed fitting params from `ALSModel`.
coderxiang
Author: Xiangrui Meng <meng@databricks.com>
Closes #6468 from mengxr/SPARK-7922 and squashes the following commits:
7bfb1d5 [Xiangrui Meng] update ALSModel in PySpark
1ba5607 [Xiangrui Meng] use DataFrames for user/item factors in ALS
Diffstat (limited to 'python/pyspark/mllib/common.py')
| -rw-r--r-- | python/pyspark/mllib/common.py | 5 |
1 files changed, 4 insertions, 1 deletions
diff --git a/python/pyspark/mllib/common.py b/python/pyspark/mllib/common.py index ba60589788..855e85f571 100644 --- a/python/pyspark/mllib/common.py +++ b/python/pyspark/mllib/common.py @@ -27,7 +27,7 @@ from py4j.java_collections import ListConverter, JavaArray, JavaList from pyspark import RDD, SparkContext from pyspark.serializers import PickleSerializer, AutoBatchedSerializer - +from pyspark.sql import DataFrame, SQLContext # Hack for support float('inf') in Py4j _old_smart_decode = py4j.protocol.smart_decode @@ -99,6 +99,9 @@ def _java2py(sc, r, encoding="bytes"): jrdd = sc._jvm.SerDe.javaToPython(r) return RDD(jrdd, sc) + if clsName == 'DataFrame': + return DataFrame(r, SQLContext(sc)) + if clsName in _picklable_classes: r = sc._jvm.SerDe.dumps(r) elif isinstance(r, (JavaArray, JavaList)): |