diff options
author | DB Tsai <dbt@netflix.com> | 2015-05-14 01:26:08 -0700 |
---|---|---|
committer | Xiangrui Meng <meng@databricks.com> | 2015-05-14 01:26:08 -0700 |
commit | c1080b6fddb22d84694da2453e46a03fbc041576 (patch) | |
tree | 53178be0b185a57a0c7b9a20ec18ac988e32dcb1 /examples/src/main/python/ml/simple_text_classification_pipeline.py | |
parent | 1b8625f4258d6d1a049d0ba60e39e9757f5a568b (diff) | |
download | spark-c1080b6fddb22d84694da2453e46a03fbc041576.tar.gz spark-c1080b6fddb22d84694da2453e46a03fbc041576.tar.bz2 spark-c1080b6fddb22d84694da2453e46a03fbc041576.zip |
[SPARK-7568] [ML] ml.LogisticRegression doesn't output the right prediction
The difference is because we previously don't fit the intercept in Spark 1.3. Here, we change the input `String` so that the probability of instance 6 can be classified as `1.0` without any ambiguity.
with lambda = 0.001 in current LOR implementation, the prediction is
```
(4, spark i j k) --> prob=[0.1596407738787411,0.8403592261212589], prediction=1.0
(5, l m n) --> prob=[0.8378325685476612,0.16216743145233883], prediction=0.0
(6, spark hadoop spark) --> prob=[0.0692663313297627,0.9307336686702373], prediction=1.0
(7, apache hadoop) --> prob=[0.9821575333444208,0.01784246665557917], prediction=0.0
```
and the training accuracy is
```
(0, a b c d e spark) --> prob=[0.0021342419881406746,0.9978657580118594], prediction=1.0
(1, b d) --> prob=[0.9959176174854043,0.004082382514595685], prediction=0.0
(2, spark f g h) --> prob=[0.0014541569986711233,0.9985458430013289], prediction=1.0
(3, hadoop mapreduce) --> prob=[0.9982978367343561,0.0017021632656438518], prediction=0.0
```
Author: DB Tsai <dbt@netflix.com>
Closes #6109 from dbtsai/lor-example and squashes the following commits:
ac63ce4 [DB Tsai] first commit
Diffstat (limited to 'examples/src/main/python/ml/simple_text_classification_pipeline.py')
-rw-r--r-- | examples/src/main/python/ml/simple_text_classification_pipeline.py | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/examples/src/main/python/ml/simple_text_classification_pipeline.py b/examples/src/main/python/ml/simple_text_classification_pipeline.py index fab21f003b..b4f06bf888 100644 --- a/examples/src/main/python/ml/simple_text_classification_pipeline.py +++ b/examples/src/main/python/ml/simple_text_classification_pipeline.py @@ -48,7 +48,7 @@ if __name__ == "__main__": # Configure an ML pipeline, which consists of tree stages: tokenizer, hashingTF, and lr. tokenizer = Tokenizer(inputCol="text", outputCol="words") hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features") - lr = LogisticRegression(maxIter=10, regParam=0.01) + lr = LogisticRegression(maxIter=10, regParam=0.001) pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) # Fit the pipeline to training documents. @@ -58,7 +58,7 @@ if __name__ == "__main__": Document = Row("id", "text") test = sc.parallelize([(4, "spark i j k"), (5, "l m n"), - (6, "mapreduce spark"), + (6, "spark hadoop spark"), (7, "apache hadoop")]) \ .map(lambda x: Document(*x)).toDF() |