diff options
| author | Michał Wesołowski <michal.wesolowski@bzwbk.pl> | 2016-07-19 12:18:42 +0100 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-07-19 12:18:42 +0100 |
| commit | 5d92326be76cb15edc6e18e94a373e197f696803 (patch) | |
| tree | f7d8bdccef69e22a825fd09b2b36ab816e5bd3f6 /dev/deps/spark-deps-hadoop-2.3 | |
| parent | 6c4b9f4be6b429197c6a53f937a82c2ac5866d65 (diff) | |
| download | spark-5d92326be76cb15edc6e18e94a373e197f696803.tar.gz spark-5d92326be76cb15edc6e18e94a373e197f696803.tar.bz2 spark-5d92326be76cb15edc6e18e94a373e197f696803.zip | |
[SPARK-16478] graphX (added graph caching in strongly connected components)
## What changes were proposed in this pull request?
I added caching in every iteration for sccGraph that is returned in strongly connected components. Without this cache strongly connected components returned graph that needed to be computed from scratch when some intermediary caches didn't existed anymore.
## How was this patch tested?
I tested it by running code similar to the one [on databrics](https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/4889410027417133/3634650767364730/3117184429335832/latest.html). Basically I generated large graph and computed strongly connected components with changed code, than simply run count on vertices and edges. Count after this update takes few seconds instead 20 minutes.
# statement
contribution is my original work and I license the work to the project under the project's open source license.
Author: Michał Wesołowski <michal.wesolowski@bzwbk.pl>
Closes #14137 from wesolowskim/SPARK-16478.
Diffstat (limited to 'dev/deps/spark-deps-hadoop-2.3')
0 files changed, 0 insertions, 0 deletions