diff options
| author | Roger Menezes <rmenezes@netflix.com> | 2015-06-12 18:29:58 -0700 |
|---|---|---|
| committer | DB Tsai <dbt@netflix.com> | 2015-06-12 18:29:58 -0700 |
| commit | 6e9c3ff1ecaf12a0126d83f27f5a4153ae420a34 (patch) | |
| tree | b2119951c52ce72f894c859c5a64918b9d3c3971 /examples/src/main | |
| parent | e9471d3414d327c7d0853e18f1844ab1bd09c8ed (diff) | |
| download | spark-6e9c3ff1ecaf12a0126d83f27f5a4153ae420a34.tar.gz spark-6e9c3ff1ecaf12a0126d83f27f5a4153ae420a34.tar.bz2 spark-6e9c3ff1ecaf12a0126d83f27f5a4153ae420a34.zip | |
[SPARK-8314][MLlib] improvement in performance of MLUtils.appendBias
MLUtils.appendBias method is heavily used in creating intercepts for linear models.
This method uses Breeze's vector concatenation which is very slow compared to the plain
System.arrayCopy. This improvement is to change the implementation to use System.arrayCopy.
I saw the following performance improvements after the change:
Benchmark with mnist dataset for 50 times:
MLUtils.appendBias (SparseVector Before): 47320 ms
MLUtils.appendBias (SparseVector After): 1935 ms
MLUtils.appendBias (DenseVector Before): 5340 ms
MLUtils.appendBias (DenseVector After): 4080 ms
This is almost a 24 times performance boost for SparseVectors.
Author: Roger Menezes <rmenezes@netflix.com>
Closes #6768 from rogermenezes/improve-append-bias and squashes the following commits:
4e42f75 [Roger Menezes] address feedback
e999d79 [Roger Menezes] first commit
Diffstat (limited to 'examples/src/main')
| -rw-r--r-- | examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala b/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala index b12f833ce9..3cf193f353 100644 --- a/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala +++ b/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala @@ -145,9 +145,9 @@ object LogisticRegressionExample { val elapsedTime = (System.nanoTime() - startTime) / 1e9 println(s"Training time: $elapsedTime seconds") - val lirModel = pipelineModel.stages.last.asInstanceOf[LogisticRegressionModel] + val lorModel = pipelineModel.stages.last.asInstanceOf[LogisticRegressionModel] // Print the weights and intercept for logistic regression. - println(s"Weights: ${lirModel.weights} Intercept: ${lirModel.intercept}") + println(s"Weights: ${lorModel.weights} Intercept: ${lorModel.intercept}") println("Training data results:") DecisionTreeExample.evaluateClassificationModel(pipelineModel, training, "indexedLabel") |