diff options
author | Xiangrui Meng <meng@databricks.com> | 2015-05-29 00:51:12 -0700 |
---|---|---|
committer | Xiangrui Meng <meng@databricks.com> | 2015-05-29 00:51:12 -0700 |
commit | 23452be944463dae72a35b58551040556dd3aeb5 (patch) | |
tree | eae02d4bd926c32d0eb9388ae031d3d329f89c43 /python/pyspark/ml/evaluation.py | |
parent | 97a60cf75d1fed654953eccedd04f3442389c5ca (diff) | |
download | spark-23452be944463dae72a35b58551040556dd3aeb5.tar.gz spark-23452be944463dae72a35b58551040556dd3aeb5.tar.bz2 spark-23452be944463dae72a35b58551040556dd3aeb5.zip |
[SPARK-7912] [SPARK-7921] [MLLIB] Update OneHotEncoder to handle ML attributes and change includeFirst to dropLast
This PR contains two major changes to `OneHotEncoder`:
1. more robust handling of ML attributes. If the input attribute is unknown, we look at the values to get the max category index
2. change `includeFirst` to `dropLast` and leave the default to `true`. There are couple benefits:
a. consistent with other tutorials of one-hot encoding (or dummy coding) (e.g., http://www.ats.ucla.edu/stat/mult_pkg/faq/general/dummy.htm)
b. keep the indices unmodified in the output vector. If we drop the first, all indices will be shifted by 1.
c. If users use `StringIndex`, the last element is the least frequent one.
Sorry for including two changes in one PR! I'll update the user guide in another PR.
jkbradley sryza
Author: Xiangrui Meng <meng@databricks.com>
Closes #6466 from mengxr/SPARK-7912 and squashes the following commits:
a280dca [Xiangrui Meng] fix tests
d8f234d [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into SPARK-7912
171b276 [Xiangrui Meng] mention the difference between our impl vs sklearn's
00dfd96 [Xiangrui Meng] update OneHotEncoder in Python
208ddad [Xiangrui Meng] update OneHotEncoder to handle ML attributes and change includeFirst to dropLast
Diffstat (limited to 'python/pyspark/ml/evaluation.py')
0 files changed, 0 insertions, 0 deletions