diff options
author | Reynold Xin <rxin@apache.org> | 2014-01-14 21:51:06 -0800 |
---|---|---|
committer | Reynold Xin <rxin@apache.org> | 2014-01-14 21:51:06 -0800 |
commit | ad294db326f57beb98f9734e2b4c45d9da1a4c89 (patch) | |
tree | da53f1491f2d10e22e818d597853300d951ddade /docs/graphx-programming-guide.md | |
parent | 74b46acdc57293c103ab5dd5af931d0d0e32c0ed (diff) | |
parent | 1210ec29454939087e5e8163aa05e853beba5d19 (diff) | |
download | spark-ad294db326f57beb98f9734e2b4c45d9da1a4c89.tar.gz spark-ad294db326f57beb98f9734e2b4c45d9da1a4c89.tar.bz2 spark-ad294db326f57beb98f9734e2b4c45d9da1a4c89.zip |
Merge pull request #431 from ankurdave/graphx-caching-doc
Describe caching and uncaching in GraphX programming guide
Diffstat (limited to 'docs/graphx-programming-guide.md')
-rw-r--r-- | docs/graphx-programming-guide.md | 11 |
1 files changed, 10 insertions, 1 deletions
diff --git a/docs/graphx-programming-guide.md b/docs/graphx-programming-guide.md index 5641f9f137..03940d836b 100644 --- a/docs/graphx-programming-guide.md +++ b/docs/graphx-programming-guide.md @@ -611,11 +611,20 @@ class GraphOps[VD, ED] { > substantial communication. If possible try expressing the same computation using the > `mapReduceTriplets` operator directly. +## Caching and Uncaching + +In Spark, RDDs are not persisted in memory by default. To avoid recomputation, they must be explicitly cached when using them multiple times (see the [Spark Programming Guide][RDD Persistence]). Graphs in GraphX behave the same way. **When using a graph multiple times, make sure to call [`Graph.cache()`][Graph.cache] on it first.** + +[RDD Persistence]: scala-programming-guide.html#rdd-persistence +[Graph.cache]: api/graphx/index.html#org.apache.spark.graphx.Graph@cache():Graph[VD,ED] + +In iterative computations, *uncaching* may also be necessary for best performance. By default, cached RDDs and graphs will remain in memory until memory pressure forces them to be evicted in LRU order. For iterative computation, intermediate results from previous iterations will fill up the cache. Though they will eventually be evicted, the unnecessary data stored in memory will slow down garbage collection. It would be more efficient to uncache intermediate results as soon as they are no longer necessary. This involves materializing (caching and forcing) a graph or RDD every iteration, uncaching all other datasets, and only using the materialized dataset in future iterations. However, because graphs are composed of multiple RDDs, it can be difficult to unpersist them correctly. **For iterative computation we recommend using the Pregel API, which correctly unpersists intermediate results.** + # Pregel API <a name="pregel"></a> Graphs are inherently recursive data-structures as properties of vertices depend on properties of -their neighbors which intern depend on properties of *their* neighbors. As a +their neighbors which in turn depend on properties of *their* neighbors. As a consequence many important graph algorithms iteratively recompute the properties of each vertex until a fixed-point condition is reached. A range of graph-parallel abstractions have been proposed to express these iterative algorithms. GraphX exposes a Pregel-like operator which is a fusion of |