diff options
author | Jason Lee <cjlee@us.ibm.com> | 2016-01-15 12:04:05 +0000 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2016-01-15 12:04:05 +0000 |
commit | d0a5c32bd05841f411a342a80c5da9f73f30d69a (patch) | |
tree | 6a53d9a4b5a038bd49837e04b29a54ce462866b7 /graphx/src/test/scala/org/apache | |
parent | fe7246fea67e1d71fba679dee3eb2c7386b4f4e2 (diff) | |
download | spark-d0a5c32bd05841f411a342a80c5da9f73f30d69a.tar.gz spark-d0a5c32bd05841f411a342a80c5da9f73f30d69a.tar.bz2 spark-d0a5c32bd05841f411a342a80c5da9f73f30d69a.zip |
[SPARK-12655][GRAPHX] GraphX does not unpersist RDDs
Some VertexRDD and EdgeRDD are created during the intermediate step of g.connectedComponents() but unnecessarily left cached after the method is done. The fix is to unpersist these RDDs once they are no longer in use.
A test case is added to confirm the fix for the reported bug.
Author: Jason Lee <cjlee@us.ibm.com>
Closes #10713 from jasoncl/SPARK-12655.
Diffstat (limited to 'graphx/src/test/scala/org/apache')
-rw-r--r-- | graphx/src/test/scala/org/apache/spark/graphx/GraphSuite.scala | 16 |
1 files changed, 16 insertions, 0 deletions
diff --git a/graphx/src/test/scala/org/apache/spark/graphx/GraphSuite.scala b/graphx/src/test/scala/org/apache/spark/graphx/GraphSuite.scala index 1f5e27d550..2fbc6f069d 100644 --- a/graphx/src/test/scala/org/apache/spark/graphx/GraphSuite.scala +++ b/graphx/src/test/scala/org/apache/spark/graphx/GraphSuite.scala @@ -428,4 +428,20 @@ class GraphSuite extends SparkFunSuite with LocalSparkContext { } } + test("unpersist graph RDD") { + withSpark { sc => + val vert = sc.parallelize(List((1L, "a"), (2L, "b"), (3L, "c")), 1) + val edges = sc.parallelize(List(Edge[Long](1L, 2L), Edge[Long](1L, 3L)), 1) + val g0 = Graph(vert, edges) + val g = g0.partitionBy(PartitionStrategy.EdgePartition2D, 2) + val cc = g.connectedComponents() + assert(sc.getPersistentRDDs.nonEmpty) + cc.unpersist() + g.unpersist() + g0.unpersist() + vert.unpersist() + edges.unpersist() + assert(sc.getPersistentRDDs.isEmpty) + } + } } |