From c3462c65684885299cf037d56c88bd53c08c6348 Mon Sep 17 00:00:00 2001 From: Gregory Owen Date: Mon, 21 Jul 2014 18:55:01 -0700 Subject: [SPARK-2086] Improve output of toDebugString to make shuffle boundaries more clear Changes RDD.toDebugString() to show hierarchy and shuffle transformations more clearly New output: ``` (3) FlatMappedValuesRDD[325] at apply at Transformer.scala:22 | MappedValuesRDD[324] at apply at Transformer.scala:22 | CoGroupedRDD[323] at apply at Transformer.scala:22 +-(5) MappedRDD[320] at apply at Transformer.scala:22 | | MappedRDD[319] at apply at Transformer.scala:22 | | MappedValuesRDD[318] at apply at Transformer.scala:22 | | MapPartitionsRDD[317] at apply at Transformer.scala:22 | | ShuffledRDD[316] at apply at Transformer.scala:22 | +-(10) MappedRDD[315] at apply at Transformer.scala:22 | | ParallelCollectionRDD[314] at apply at Transformer.scala:22 +-(100) MappedRDD[322] at apply at Transformer.scala:22 | ParallelCollectionRDD[321] at apply at Transformer.scala:22 ``` Author: Gregory Owen Closes #1364 from GregOwen/to-debug-string and squashes the following commits: 08f5c78 [Gregory Owen] toDebugString: prettier debug printing to show shuffles and joins more clearly 1603f7b [Gregory Owen] toDebugString: prettier debug printing to show shuffles and joins more clearly --- project/MimaExcludes.scala | 8 ++++++++ 1 file changed, 8 insertions(+) (limited to 'project/MimaExcludes.scala') diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala index e0f433b26f..4d86e1a0d8 100644 --- a/project/MimaExcludes.scala +++ b/project/MimaExcludes.scala @@ -61,6 +61,14 @@ object MimaExcludes { "org.apache.spark.api.java.JavaDoubleRDD.countApproxDistinct$default$1"), ProblemFilters.exclude[MissingMethodProblem]( "org.apache.spark.storage.MemoryStore.Entry"), + ProblemFilters.exclude[MissingMethodProblem]( + "org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$debugChildren$1"), + ProblemFilters.exclude[MissingMethodProblem]( + "org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$firstDebugString$1"), + ProblemFilters.exclude[MissingMethodProblem]( + "org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$shuffleDebugString$1"), + ProblemFilters.exclude[MissingMethodProblem]( + "org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$debugString$1"), ProblemFilters.exclude[MissingMethodProblem]( "org.apache.spark.rdd.PairRDDFunctions.org$apache$spark$rdd$PairRDDFunctions$$" + "createZero$1") -- cgit v1.2.3