diff options
author | WeichenXu <WeichenXu123@outlook.com> | 2016-07-12 13:04:34 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2016-07-12 13:04:34 +0100 |
commit | 6cb75db9ab1a4f227069bec2763b89546b88b0ee (patch) | |
tree | 70fb845760481b734132a27d7c1f68a93c092eeb | |
parent | 5b28e02584fa4da85214e7da6d77b3b8e189b781 (diff) | |
download | spark-6cb75db9ab1a4f227069bec2763b89546b88b0ee.tar.gz spark-6cb75db9ab1a4f227069bec2763b89546b88b0ee.tar.bz2 spark-6cb75db9ab1a4f227069bec2763b89546b88b0ee.zip |
[SPARK-16470][ML][OPTIMIZER] Check linear regression training whether actually reach convergence and add warning if not
## What changes were proposed in this pull request?
In `ml.regression.LinearRegression`, it use breeze `LBFGS` and `OWLQN` optimizer to do data training, but do not check whether breeze's optimizer returned result actually reached convergence.
The `LBFGS` and `OWLQN` optimizer in breeze finish iteration may result the following situations:
1) reach max iteration number
2) function reach value convergence
3) objective function stop improving
4) gradient reach convergence
5) search failed(due to some internal numerical error)
I add warning printing code so that
if the iteration result is (1) or (3) or (5) in above, it will print a warning with respective reason string.
## How was this patch tested?
Manual.
Author: WeichenXu <WeichenXu123@outlook.com>
Closes #14122 from WeichenXu123/add_lr_not_convergence_warn.
-rw-r--r-- | mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala index 0477f71f32..6b82ae14e1 100644 --- a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala +++ b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala @@ -327,6 +327,11 @@ class LinearRegression @Since("1.3.0") (@Since("1.3.0") override val uid: String throw new SparkException(msg) } + if (!state.actuallyConverged) { + logWarning("LinearRegression training fininshed but the result " + + s"is not converged because: ${state.convergedReason.get.reason}") + } + /* The coefficients are trained in the scaled space; we're converting them back to the original space. |