diff options
| author | Yong Gang Cao <ygcao@amazon.com> | 2016-02-22 09:47:36 +0000 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-02-22 09:47:36 +0000 |
| commit | ef1047fca789e5470b7b12974f0435d6d1c4f2d5 (patch) | |
| tree | 9c9e646565e0b7603b0da5b7f272406acf78a8a6 /python/pyspark/ml/feature.py | |
| parent | 8f35d3eac9268127512851e52864e64b0bae2f33 (diff) | |
| download | spark-ef1047fca789e5470b7b12974f0435d6d1c4f2d5.tar.gz spark-ef1047fca789e5470b7b12974f0435d6d1c4f2d5.tar.bz2 spark-ef1047fca789e5470b7b12974f0435d6d1c4f2d5.zip | |
[SPARK-12153][SPARK-7617][MLLIB] add support of arbitrary length sentence and other tuning for Word2Vec
add support of arbitrary length sentence by using the nature representation of sentences in the input.
add new similarity functions and add normalization option for distances in synonym finding
add new accessor for internal structure(the vocabulary and wordindex) for convenience
need instructions about how to set value for the Since annotation for newly added public functions. 1.5.3?
jira link: https://issues.apache.org/jira/browse/SPARK-12153
Author: Yong Gang Cao <ygcao@amazon.com>
Author: Yong-Gang Cao <ygcao@users.noreply.github.com>
Closes #10152 from ygcao/improvementForSentenceBoundary.
Diffstat (limited to 'python/pyspark/ml/feature.py')
| -rw-r--r-- | python/pyspark/ml/feature.py | 12 |
1 files changed, 6 insertions, 6 deletions
diff --git a/python/pyspark/ml/feature.py b/python/pyspark/ml/feature.py index d017a23188..464c9446f2 100644 --- a/python/pyspark/ml/feature.py +++ b/python/pyspark/ml/feature.py @@ -1836,12 +1836,12 @@ class Word2Vec(JavaEstimator, HasStepSize, HasMaxIter, HasSeed, HasInputCol, Has +----+--------------------+ ... >>> model.findSynonyms("a", 2).show() - +----+--------------------+ - |word| similarity| - +----+--------------------+ - | b| 0.16782984556103436| - | c|-0.46761559092107646| - +----+--------------------+ + +----+-------------------+ + |word| similarity| + +----+-------------------+ + | b| 0.2505344027513247| + | c|-0.6980510075367647| + +----+-------------------+ ... >>> model.transform(doc).head().model DenseVector([0.5524, -0.4995, -0.3599, 0.0241, 0.3461]) |