diff options
author | Yanbo Liang <ybliang8@gmail.com> | 2016-08-19 03:23:16 -0700 |
---|---|---|
committer | Yanbo Liang <ybliang8@gmail.com> | 2016-08-19 03:23:16 -0700 |
commit | 864be9359ae2f8409e6dbc38a7a18593f9cc5692 (patch) | |
tree | 0995685df71db57dd6360fb3830fe7364b6fb42c /mllib/src/test | |
parent | 5377fc62360d5e9b5c94078e41d10a96e0e8a535 (diff) | |
download | spark-864be9359ae2f8409e6dbc38a7a18593f9cc5692.tar.gz spark-864be9359ae2f8409e6dbc38a7a18593f9cc5692.tar.bz2 spark-864be9359ae2f8409e6dbc38a7a18593f9cc5692.zip |
[SPARK-17141][ML] MinMaxScaler should remain NaN value.
## What changes were proposed in this pull request?
In the existing code, ```MinMaxScaler``` handle ```NaN``` value indeterminately.
* If a column has identity value, that is ```max == min```, ```MinMaxScalerModel``` transformation will output ```0.5``` for all rows even the original value is ```NaN```.
* Otherwise, it will remain ```NaN``` after transformation.
I think we should unify the behavior by remaining ```NaN``` value at any condition, since we don't know how to transform a ```NaN``` value. In Python sklearn, it will throw exception when there is ```NaN``` in the dataset.
## How was this patch tested?
Unit tests.
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #14716 from yanboliang/spark-17141.
Diffstat (limited to 'mllib/src/test')
-rw-r--r-- | mllib/src/test/scala/org/apache/spark/ml/feature/MinMaxScalerSuite.scala | 27 |
1 files changed, 27 insertions, 0 deletions
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/MinMaxScalerSuite.scala b/mllib/src/test/scala/org/apache/spark/ml/feature/MinMaxScalerSuite.scala index 5da8471175..9f376b7003 100644 --- a/mllib/src/test/scala/org/apache/spark/ml/feature/MinMaxScalerSuite.scala +++ b/mllib/src/test/scala/org/apache/spark/ml/feature/MinMaxScalerSuite.scala @@ -90,4 +90,31 @@ class MinMaxScalerSuite extends SparkFunSuite with MLlibTestSparkContext with De assert(newInstance.originalMin === instance.originalMin) assert(newInstance.originalMax === instance.originalMax) } + + test("MinMaxScaler should remain NaN value") { + val data = Array( + Vectors.dense(1, Double.NaN, 2.0, 2.0), + Vectors.dense(2, 2.0, 0.0, 3.0), + Vectors.dense(3, Double.NaN, 0.0, 1.0), + Vectors.dense(6, 2.0, 2.0, Double.NaN)) + + val expected: Array[Vector] = Array( + Vectors.dense(-5.0, Double.NaN, 5.0, 0.0), + Vectors.dense(-3.0, 0.0, -5.0, 5.0), + Vectors.dense(-1.0, Double.NaN, -5.0, -5.0), + Vectors.dense(5.0, 0.0, 5.0, Double.NaN)) + + val df = spark.createDataFrame(data.zip(expected)).toDF("features", "expected") + val scaler = new MinMaxScaler() + .setInputCol("features") + .setOutputCol("scaled") + .setMin(-5) + .setMax(5) + + val model = scaler.fit(df) + model.transform(df).select("expected", "scaled").collect() + .foreach { case Row(vector1: Vector, vector2: Vector) => + assert(vector1.equals(vector2), "Transformed vector is different with expected.") + } + } } |