diff options
| author | Yuhao Yang <hhbyyh@gmail.com> | 2016-03-30 15:58:19 -0700 |
|---|---|---|
| committer | Joseph K. Bradley <joseph@databricks.com> | 2016-03-30 15:58:19 -0700 |
| commit | ca458618d8ee659ffa9a081083cd475a440fa8ff (patch) | |
| tree | 117c2da92f8aefb210f8b9bd0c4736fbb023248c /yarn | |
| parent | f301df37cb63aeecf48077ae56351538e6eeeeb7 (diff) | |
| download | spark-ca458618d8ee659ffa9a081083cd475a440fa8ff.tar.gz spark-ca458618d8ee659ffa9a081083cd475a440fa8ff.tar.bz2 spark-ca458618d8ee659ffa9a081083cd475a440fa8ff.zip | |
[SPARK-11507][MLLIB] add compact in Matrices fromBreeze
jira: https://issues.apache.org/jira/browse/SPARK-11507
"In certain situations when adding two block matrices, I get an error regarding colPtr and the operation fails. External issue URL includes full error and code for reproducing the problem."
root cause: colPtr.last does NOT always equal to values.length in breeze SCSMatrix, which fails the require in SparseMatrix.
easy step to repro:
```
val m1: BM[Double] = new CSCMatrix[Double] (Array (1.0, 1, 1), 3, 3, Array (0, 1, 2, 3), Array (0, 1, 2) )
val m2: BM[Double] = new CSCMatrix[Double] (Array (1.0, 2, 2, 4), 3, 3, Array (0, 0, 2, 4), Array (1, 2, 1, 2) )
val sum = m1 + m2
Matrices.fromBreeze(sum)
```
Solution: By checking the code in [CSCMatrix](https://github.com/scalanlp/breeze/blob/28000a7b901bc3cfbbbf5c0bce1d0a5dda8281b0/math/src/main/scala/breeze/linalg/CSCMatrix.scala), CSCMatrix in breeze can have extra zeros in the end of data array. Invoking compact will make sure it aligns with the require of SparseMatrix. This should add limited overhead as the actual compact operation is only performed when necessary.
Author: Yuhao Yang <hhbyyh@gmail.com>
Closes #9520 from hhbyyh/matricesFromBreeze.
Diffstat (limited to 'yarn')
0 files changed, 0 insertions, 0 deletions