diff options
| author | Yuhao Yang <hhbyyh@gmail.com> | 2015-08-01 02:31:28 -0700 |
|---|---|---|
| committer | Xiangrui Meng <meng@databricks.com> | 2015-08-01 02:31:28 -0700 |
| commit | 8765665015ef47a23e00f7d01d4d280c31bb236d (patch) | |
| tree | ec5107a65819712061aa4c3337f79367bc088519 /sql | |
| parent | d2a9b66f6c0de89d6d16370af1c77c7f51b11d3e (diff) | |
| download | spark-8765665015ef47a23e00f7d01d4d280c31bb236d.tar.gz spark-8765665015ef47a23e00f7d01d4d280c31bb236d.tar.bz2 spark-8765665015ef47a23e00f7d01d4d280c31bb236d.zip | |
[SPARK-8169] [ML] Add StopWordsRemover as a transformer
jira: https://issues.apache.org/jira/browse/SPARK-8169
stop words: http://en.wikipedia.org/wiki/Stop_words
StopWordsRemover takes a string array column and outputs a string array column with all defined stop words removed. The transformer should also come with a standard set of stop words as default.
Currently I used a minimum stop words set since on some [case](http://nlp.stanford.edu/IR-book/html/htmledition/dropping-common-terms-stop-words-1.html), small set of stop words is preferred.
ASCII char has been tested, Yet I cannot check it in due to style check.
Further thought,
1. Maybe I should use OpenHashSet. Is it recommended?
2. Currently I leave the null in input array untouched, i.e. Array(null, null) => Array(null, null).
3. If the current stop words set looks too limited, any suggestion for replacement? We can have something similar to the one in [SKlearn](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/stop_words.py).
Author: Yuhao Yang <hhbyyh@gmail.com>
Closes #6742 from hhbyyh/stopwords and squashes the following commits:
fa959d8 [Yuhao Yang] separating udf
f190217 [Yuhao Yang] replace default list and other small fix
04403ab [Yuhao Yang] Merge remote-tracking branch 'upstream/master' into stopwords
b3aa957 [Yuhao Yang] add stopWordsRemover
Diffstat (limited to 'sql')
0 files changed, 0 insertions, 0 deletions