aboutsummaryrefslogtreecommitdiff
path: root/docs/graphx-programming-guide.md
diff options
context:
space:
mode:
authorAnkur Dave <ankurdave@gmail.com>2014-01-14 17:24:25 -0800
committerAnkur Dave <ankurdave@gmail.com>2014-01-14 17:25:38 -0800
commit1210ec29454939087e5e8163aa05e853beba5d19 (patch)
treeda53f1491f2d10e22e818d597853300d951ddade /docs/graphx-programming-guide.md
parent74b46acdc57293c103ab5dd5af931d0d0e32c0ed (diff)
downloadspark-1210ec29454939087e5e8163aa05e853beba5d19.tar.gz
spark-1210ec29454939087e5e8163aa05e853beba5d19.tar.bz2
spark-1210ec29454939087e5e8163aa05e853beba5d19.zip
Describe GraphX caching and uncaching in guide
Diffstat (limited to 'docs/graphx-programming-guide.md')
-rw-r--r--docs/graphx-programming-guide.md11
1 files changed, 10 insertions, 1 deletions
diff --git a/docs/graphx-programming-guide.md b/docs/graphx-programming-guide.md
index 5641f9f137..03940d836b 100644
--- a/docs/graphx-programming-guide.md
+++ b/docs/graphx-programming-guide.md
@@ -611,11 +611,20 @@ class GraphOps[VD, ED] {
> substantial communication. If possible try expressing the same computation using the
> `mapReduceTriplets` operator directly.
+## Caching and Uncaching
+
+In Spark, RDDs are not persisted in memory by default. To avoid recomputation, they must be explicitly cached when using them multiple times (see the [Spark Programming Guide][RDD Persistence]). Graphs in GraphX behave the same way. **When using a graph multiple times, make sure to call [`Graph.cache()`][Graph.cache] on it first.**
+
+[RDD Persistence]: scala-programming-guide.html#rdd-persistence
+[Graph.cache]: api/graphx/index.html#org.apache.spark.graphx.Graph@cache():Graph[VD,ED]
+
+In iterative computations, *uncaching* may also be necessary for best performance. By default, cached RDDs and graphs will remain in memory until memory pressure forces them to be evicted in LRU order. For iterative computation, intermediate results from previous iterations will fill up the cache. Though they will eventually be evicted, the unnecessary data stored in memory will slow down garbage collection. It would be more efficient to uncache intermediate results as soon as they are no longer necessary. This involves materializing (caching and forcing) a graph or RDD every iteration, uncaching all other datasets, and only using the materialized dataset in future iterations. However, because graphs are composed of multiple RDDs, it can be difficult to unpersist them correctly. **For iterative computation we recommend using the Pregel API, which correctly unpersists intermediate results.**
+
# Pregel API
<a name="pregel"></a>
Graphs are inherently recursive data-structures as properties of vertices depend on properties of
-their neighbors which intern depend on properties of *their* neighbors. As a
+their neighbors which in turn depend on properties of *their* neighbors. As a
consequence many important graph algorithms iteratively recompute the properties of each vertex
until a fixed-point condition is reached. A range of graph-parallel abstractions have been proposed
to express these iterative algorithms. GraphX exposes a Pregel-like operator which is a fusion of