diff options
| author | Xiangrui Meng <meng@databricks.com> | 2014-07-31 21:14:08 -0700 |
|---|---|---|
| committer | Xiangrui Meng <meng@databricks.com> | 2014-07-31 21:14:08 -0700 |
| commit | b19008320bdf7064e764db04c43ef003a3ce0ecd (patch) | |
| tree | ea2bf2dfd5a6dcd77b4bbc59fb59ef3997914fcb /mllib | |
| parent | 9998efab96a4fdc927818eaae53c04f946c4cf13 (diff) | |
| download | spark-b19008320bdf7064e764db04c43ef003a3ce0ecd.tar.gz spark-b19008320bdf7064e764db04c43ef003a3ce0ecd.tar.bz2 spark-b19008320bdf7064e764db04c43ef003a3ce0ecd.zip | |
[SPARK-2777][MLLIB] change ALS factors storage level to MEMORY_AND_DISK
Now the factors are persisted in memory only. If they get kicked off by later jobs, we might have to start the computation from very beginning. A better solution is changing the storage level to `MEMORY_AND_DISK`.
srowen
Author: Xiangrui Meng <meng@databricks.com>
Closes #1700 from mengxr/als-level and squashes the following commits:
c103d76 [Xiangrui Meng] change ALS factors storage level to MEMORY_AND_DISK
Diffstat (limited to 'mllib')
| -rw-r--r-- | mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala b/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala index d208cfb917..36d262fed4 100644 --- a/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala +++ b/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala @@ -290,8 +290,8 @@ class ALS private ( val usersOut = unblockFactors(users, userOutLinks) val productsOut = unblockFactors(products, productOutLinks) - usersOut.setName("usersOut").persist() - productsOut.setName("productsOut").persist() + usersOut.setName("usersOut").persist(StorageLevel.MEMORY_AND_DISK) + productsOut.setName("productsOut").persist(StorageLevel.MEMORY_AND_DISK) // Materialize usersOut and productsOut. usersOut.count() |