diff options
| author | hyukjinkwon <gurwls223@gmail.com> | 2016-05-19 08:52:41 +0200 |
|---|---|---|
| committer | Nick Pentreath <nickp@za.ibm.com> | 2016-05-19 08:52:41 +0200 |
| commit | e2ec32dab8530aa21ec95a27d60b1c22f3d1a18c (patch) | |
| tree | 6c6f9fa78578b1a23f954c27beeb509c203098b0 /examples/src/main/scala | |
| parent | 661c21049b62ebfaf788dcbc31d62a09e206265b (diff) | |
| download | spark-e2ec32dab8530aa21ec95a27d60b1c22f3d1a18c.tar.gz spark-e2ec32dab8530aa21ec95a27d60b1c22f3d1a18c.tar.bz2 spark-e2ec32dab8530aa21ec95a27d60b1c22f3d1a18c.zip | |
[SPARK-15031][EXAMPLES][FOLLOW-UP] Make Python param example working with SparkSession
## What changes were proposed in this pull request?
It seems most of Python examples were changed to use SparkSession by https://github.com/apache/spark/pull/12809. This PR said both examples below:
- `simple_params_example.py`
- `aft_survival_regression.py`
are not changed because it dose not work. It seems `aft_survival_regression.py` is changed by https://github.com/apache/spark/pull/13050 but `simple_params_example.py` is not yet.
This PR corrects the example and make this use SparkSession.
In more detail, it seems `threshold` is replaced to `thresholds` here and there by https://github.com/apache/spark/commit/5a23213c148bfe362514f9c71f5273ebda0a848a. However, when it calls `lr.fit(training, paramMap)` this overwrites the values. So, `threshold` was 5 and `thresholds` becomes 5.5 (by `1 / (1 + thresholds(0) / thresholds(1)`).
According to the comment below. this is not allowed, https://github.com/apache/spark/blob/354f8f11bd4b20fa99bd67a98da3525fd3d75c81/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala#L58-L61.
So, in this PR, it sets the equivalent value so that this does not throw an exception.
## How was this patch tested?
Manully (`mvn package -DskipTests && spark-submit simple_params_example.py`)
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #13135 from HyukjinKwon/SPARK-15031.
Diffstat (limited to 'examples/src/main/scala')
| -rw-r--r-- | examples/src/main/scala/org/apache/spark/examples/ml/SimpleParamsExample.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/examples/src/main/scala/org/apache/spark/examples/ml/SimpleParamsExample.scala b/examples/src/main/scala/org/apache/spark/examples/ml/SimpleParamsExample.scala index 3355c8ffa2..29f1f50960 100644 --- a/examples/src/main/scala/org/apache/spark/examples/ml/SimpleParamsExample.scala +++ b/examples/src/main/scala/org/apache/spark/examples/ml/SimpleParamsExample.scala @@ -70,7 +70,7 @@ object SimpleParamsExample { // which supports several methods for specifying parameters. val paramMap = ParamMap(lr.maxIter -> 20) paramMap.put(lr.maxIter, 30) // Specify 1 Param. This overwrites the original maxIter. - paramMap.put(lr.regParam -> 0.1, lr.thresholds -> Array(0.45, 0.55)) // Specify multiple Params. + paramMap.put(lr.regParam -> 0.1, lr.thresholds -> Array(0.5, 0.5)) // Specify multiple Params. // One can also combine ParamMaps. val paramMap2 = ParamMap(lr.probabilityCol -> "myProbability") // Change output column name |