diff options
| author | qiping.lqp <qiping.lqp@alibaba-inc.com> | 2014-10-03 03:26:17 -0700 |
|---|---|---|
| committer | Xiangrui Meng <meng@databricks.com> | 2014-10-03 03:26:17 -0700 |
| commit | 2e4eae3a52e3d04895b00447d1ac56ae3c1b98ae (patch) | |
| tree | 58acf9a7570d29de3d353383c3de1e428a056e9c /project | |
| parent | 1c90347a4bba12df7b76d282a7dbac8e555e049f (diff) | |
| download | spark-2e4eae3a52e3d04895b00447d1ac56ae3c1b98ae.tar.gz spark-2e4eae3a52e3d04895b00447d1ac56ae3c1b98ae.tar.bz2 spark-2e4eae3a52e3d04895b00447d1ac56ae3c1b98ae.zip | |
[SPARK-3366][MLLIB]Compute best splits distributively in decision tree
Currently, all best splits are computed on the driver, which makes the driver a bottleneck for both communication and computation. This PR fix this problem by computed best splits on executors.
Instead of send all aggregate stats to the driver node, we can send aggregate stats for a node to a particular executor, using `reduceByKey` operation, then we can compute best split for this node there.
Implementation details:
Each node now has a nodeStatsAggregator, which save aggregate stats for all features and bins.
First use mapPartition to compute node aggregate stats for all nodes in each partition.
Then transform node aggregate stats to (nodeIndex, nodeStatsAggregator) pairs and use to `reduceByKey` operation to combine nodeStatsAggregator for the same node.
After all stats have been combined, best splits can be computed for each node based on the node aggregate stats. Best split result is collected to driver to construct the decision tree.
CC: mengxr manishamde jkbradley, please help me review this, thanks.
Author: qiping.lqp <qiping.lqp@alibaba-inc.com>
Author: chouqin <liqiping1991@gmail.com>
Closes #2595 from chouqin/dt-dist-agg and squashes the following commits:
db0d24a [chouqin] fix a minor bug and adjust code
a0d9de3 [chouqin] adjust code based on comments
9f201a6 [chouqin] fix bug: statsSize -> allStatsSize
a8a7ed0 [chouqin] Merge branch 'master' of https://github.com/apache/spark into dt-dist-agg
f13b346 [chouqin] adjust randomforest comments
c32636e [chouqin] adjust code based on comments
ac6a505 [chouqin] adjust code based on comments
7bbb787 [chouqin] add comments
bdd2a63 [qiping.lqp] fix test suite
a75df27 [qiping.lqp] fix test suite
b5b0bc2 [qiping.lqp] fix style
e76414f [qiping.lqp] fix testsuite
748bd45 [qiping.lqp] fix type-mismatch bug
24eacd8 [qiping.lqp] fix type-mismatch bug
5f63d6c [qiping.lqp] add multiclassification using One-Vs-All strategy
4f56496 [qiping.lqp] fix bug
f00fc22 [qiping.lqp] fix bug
532993a [qiping.lqp] Compute best splits distributively in decision tree
Diffstat (limited to 'project')
0 files changed, 0 insertions, 0 deletions