diff options
author | Gregory Owen <greowen@gmail.com> | 2014-07-21 18:55:01 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-07-21 18:55:01 -0700 |
commit | c3462c65684885299cf037d56c88bd53c08c6348 (patch) | |
tree | f975d0237f8a6d4edb88e47c3dfa25ccc46ac133 /python/pyspark/join.py | |
parent | 511a7314037219c23e824ea5363bf7f1df55bab3 (diff) | |
download | spark-c3462c65684885299cf037d56c88bd53c08c6348.tar.gz spark-c3462c65684885299cf037d56c88bd53c08c6348.tar.bz2 spark-c3462c65684885299cf037d56c88bd53c08c6348.zip |
[SPARK-2086] Improve output of toDebugString to make shuffle boundaries more clear
Changes RDD.toDebugString() to show hierarchy and shuffle transformations more clearly
New output:
```
(3) FlatMappedValuesRDD[325] at apply at Transformer.scala:22
| MappedValuesRDD[324] at apply at Transformer.scala:22
| CoGroupedRDD[323] at apply at Transformer.scala:22
+-(5) MappedRDD[320] at apply at Transformer.scala:22
| | MappedRDD[319] at apply at Transformer.scala:22
| | MappedValuesRDD[318] at apply at Transformer.scala:22
| | MapPartitionsRDD[317] at apply at Transformer.scala:22
| | ShuffledRDD[316] at apply at Transformer.scala:22
| +-(10) MappedRDD[315] at apply at Transformer.scala:22
| | ParallelCollectionRDD[314] at apply at Transformer.scala:22
+-(100) MappedRDD[322] at apply at Transformer.scala:22
| ParallelCollectionRDD[321] at apply at Transformer.scala:22
```
Author: Gregory Owen <greowen@gmail.com>
Closes #1364 from GregOwen/to-debug-string and squashes the following commits:
08f5c78 [Gregory Owen] toDebugString: prettier debug printing to show shuffles and joins more clearly
1603f7b [Gregory Owen] toDebugString: prettier debug printing to show shuffles and joins more clearly
Diffstat (limited to 'python/pyspark/join.py')
0 files changed, 0 insertions, 0 deletions