diff options
| author | Xusen Yin <yinxusen@gmail.com> | 2015-12-07 13:16:47 -0800 |
|---|---|---|
| committer | Joseph K. Bradley <joseph@databricks.com> | 2015-12-07 13:16:47 -0800 |
| commit | 871e85d9c14c6b19068cc732951a8ae8db61b411 (patch) | |
| tree | 43dab0f6f405c465ce604ba1a95c0a26ca7efe5d /docs/ml-features.md | |
| parent | 3f4efb5c23b029496b112760fa062ff070c20334 (diff) | |
| download | spark-871e85d9c14c6b19068cc732951a8ae8db61b411.tar.gz spark-871e85d9c14c6b19068cc732951a8ae8db61b411.tar.bz2 spark-871e85d9c14c6b19068cc732951a8ae8db61b411.zip | |
[SPARK-11963][DOC] Add docs for QuantileDiscretizer
https://issues.apache.org/jira/browse/SPARK-11963
Author: Xusen Yin <yinxusen@gmail.com>
Closes #9962 from yinxusen/SPARK-11963.
Diffstat (limited to 'docs/ml-features.md')
| -rw-r--r-- | docs/ml-features.md | 65 |
1 files changed, 65 insertions, 0 deletions
diff --git a/docs/ml-features.md b/docs/ml-features.md index 05c2c96c5e..b499d6ec3b 100644 --- a/docs/ml-features.md +++ b/docs/ml-features.md @@ -1705,6 +1705,71 @@ print(output.select("features", "clicked").first()) </div> </div> +## QuantileDiscretizer + +`QuantileDiscretizer` takes a column with continuous features and outputs a column with binned +categorical features. +The bin ranges are chosen by taking a sample of the data and dividing it into roughly equal parts. +The lower and upper bin bounds will be `-Infinity` and `+Infinity`, covering all real values. +This attempts to find `numBuckets` partitions based on a sample of the given input data, but it may +find fewer depending on the data sample values. + +Note that the result may be different every time you run it, since the sample strategy behind it is +non-deterministic. + +**Examples** + +Assume that we have a DataFrame with the columns `id`, `hour`: + +~~~ + id | hour +----|------ + 0 | 18.0 +----|------ + 1 | 19.0 +----|------ + 2 | 8.0 +----|------ + 3 | 5.0 +----|------ + 4 | 2.2 +~~~ + +`hour` is a continuous feature with `Double` type. We want to turn the continuous feature into +categorical one. Given `numBuckets = 3`, we should get the following DataFrame: + +~~~ + id | hour | result +----|------|------ + 0 | 18.0 | 2.0 +----|------|------ + 1 | 19.0 | 2.0 +----|------|------ + 2 | 8.0 | 1.0 +----|------|------ + 3 | 5.0 | 1.0 +----|------|------ + 4 | 2.2 | 0.0 +~~~ + +<div class="codetabs"> +<div data-lang="scala" markdown="1"> + +Refer to the [QuantileDiscretizer Scala docs](api/scala/index.html#org.apache.spark.ml.feature.QuantileDiscretizer) +for more details on the API. + +{% include_example scala/org/apache/spark/examples/ml/QuantileDiscretizerExample.scala %} +</div> + +<div data-lang="java" markdown="1"> + +Refer to the [QuantileDiscretizer Java docs](api/java/org/apache/spark/ml/feature/QuantileDiscretizer.html) +for more details on the API. + +{% include_example java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java %} +</div> +</div> + # Feature Selectors ## VectorSlicer |