diff options
| author | Yuhao <yuhao.yang@intel.com> | 2016-12-07 05:12:24 +0800 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-12-07 05:12:24 +0800 |
| commit | fac5b75b74b2d76b6314c69be3c769f1f321688c (patch) | |
| tree | e926e5d43b0cf97e258ec8b919c781eb9c0fb9b2 /repl | |
| parent | 1ef6b296d7cd2d93cdfd5f54940842d6bb915ce0 (diff) | |
| download | spark-fac5b75b74b2d76b6314c69be3c769f1f321688c.tar.gz spark-fac5b75b74b2d76b6314c69be3c769f1f321688c.tar.bz2 spark-fac5b75b74b2d76b6314c69be3c769f1f321688c.zip | |
[SPARK-18374][ML] Incorrect words in StopWords/english.txt
## What changes were proposed in this pull request?
Currently English stop words list in MLlib contains only the argumented words after removing all the apostrophes, so "wouldn't" become "wouldn" and "t". Yet by default Tokenizer and RegexTokenizer don't split on apostrophes or quotes.
Adding original form to stop words list to match the behavior of Tokenizer and StopwordsRemover. Also remove "won" from list.
see more discussion in the jira: https://issues.apache.org/jira/browse/SPARK-18374
## How was this patch tested?
existing ut
Author: Yuhao <yuhao.yang@intel.com>
Author: Yuhao Yang <hhbyyh@gmail.com>
Closes #16103 from hhbyyh/addstopwords.
Diffstat (limited to 'repl')
0 files changed, 0 insertions, 0 deletions