aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authording <ding@localhost.localdomain>2016-10-04 00:00:10 -0700
committerJoseph K. Bradley <joseph@databricks.com>2016-10-04 00:00:10 -0700
commit126baa8d32bc0e7bf8b43f9efa84f2728f02347d (patch)
tree9aaa195bd73b5958ebf65395d0b25fe520e0d10f
parentd2dc8c4a162834818190ffd82894522c524ca3e5 (diff)
downloadspark-126baa8d32bc0e7bf8b43f9efa84f2728f02347d.tar.gz
spark-126baa8d32bc0e7bf8b43f9efa84f2728f02347d.tar.bz2
spark-126baa8d32bc0e7bf8b43f9efa84f2728f02347d.zip
[SPARK-17559][MLLIB] persist edges if their storage level is non in PeriodicGraphCheckpointer
## What changes were proposed in this pull request? When use PeriodicGraphCheckpointer to persist graph, sometimes the edges isn't persisted. As currently only when vertices's storage level is none, graph is persisted. However there is a chance vertices's storage level is not none while edges's is none. Eg. graph created by a outerJoinVertices operation, vertices is automatically cached while edges is not. In this way, edges will not be persisted if we use PeriodicGraphCheckpointer do persist. We need separately check edges's storage level and persisted it if it's none. ## How was this patch tested? manual tests Author: ding <ding@localhost.localdomain> Closes #15124 from dding3/spark-persisitEdge.
-rw-r--r--mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala5
1 files changed, 4 insertions, 1 deletions
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala b/mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala
index 20db6084d0..8007489756 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala
@@ -87,7 +87,10 @@ private[mllib] class PeriodicGraphCheckpointer[VD, ED](
override protected def persist(data: Graph[VD, ED]): Unit = {
if (data.vertices.getStorageLevel == StorageLevel.NONE) {
- data.persist()
+ data.vertices.persist()
+ }
+ if (data.edges.getStorageLevel == StorageLevel.NONE) {
+ data.edges.persist()
}
}