diff options
| author | Anthony Truchet <a.truchet@criteo.com> | 2016-07-20 10:39:59 +0100 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-07-20 10:39:59 +0100 |
| commit | 0dc79ffd1cbb45e69a35e3f5334c9a13290037a0 (patch) | |
| tree | 4363dd30ee5ee327fa62d975e33e28373ea8bbaa /mllib/src | |
| parent | 75146be6ba5e9f559f5f15430310bb476ee0812c (diff) | |
| download | spark-0dc79ffd1cbb45e69a35e3f5334c9a13290037a0.tar.gz spark-0dc79ffd1cbb45e69a35e3f5334c9a13290037a0.tar.bz2 spark-0dc79ffd1cbb45e69a35e3f5334c9a13290037a0.zip | |
[SPARK-16440][MLLIB] Destroy broadcasted variables even on driver
## What changes were proposed in this pull request?
Forgotten broadcasted variables were persisted into a previous #PR 14153). This PR turns those `unpersist()` into `destroy()` so that memory is freed even on the driver.
## How was this patch tested?
Unit Tests in Word2VecSuite were run locally.
This contribution is done on behalf of Criteo, according to the
terms of the Apache license 2.0.
Author: Anthony Truchet <a.truchet@criteo.com>
Closes #14268 from AnthonyTruchet/SPARK-16440.
Diffstat (limited to 'mllib/src')
| -rw-r--r-- | mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala | 6 |
1 files changed, 3 insertions, 3 deletions
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala b/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala index 6b9c8ee2e3..bc75646d53 100644 --- a/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala +++ b/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala @@ -434,9 +434,9 @@ class Word2Vec extends Serializable with Logging { bcSyn1Global.unpersist(false) } newSentences.unpersist() - expTable.unpersist() - bcVocab.unpersist() - bcVocabHash.unpersist() + expTable.destroy() + bcVocab.destroy() + bcVocabHash.destroy() val wordArray = vocab.map(_.word) new Word2VecModel(wordArray.zipWithIndex.toMap, syn0Global) |