diff options
| author | sethah <seth.hendrickson16@gmail.com> | 2016-08-08 00:00:15 -0700 |
|---|---|---|
| committer | DB Tsai <dbt@netflix.com> | 2016-08-08 00:00:15 -0700 |
| commit | 1db1c6567bae0c80fdc522f2cbb65557cd62263f (patch) | |
| tree | 493f86413f3e7fe5248b95fb270aee5a7739be32 /docs/streaming-custom-receivers.md | |
| parent | e076fb05ac83a3ed6995e29bb03ea07ea05e39db (diff) | |
| download | spark-1db1c6567bae0c80fdc522f2cbb65557cd62263f.tar.gz spark-1db1c6567bae0c80fdc522f2cbb65557cd62263f.tar.bz2 spark-1db1c6567bae0c80fdc522f2cbb65557cd62263f.zip | |
[SPARK-16404][ML] LeastSquaresAggregators serializes unnecessary data
## What changes were proposed in this pull request?
Similar to `LogisticAggregator`, `LeastSquaresAggregator` used for linear regression ends up serializing the coefficients and the features standard deviations, which is not necessary and can cause performance issues for high dimensional data. This patch removes this serialization.
In https://github.com/apache/spark/pull/13729 the approach was to pass these values directly to the add method. The approach used here, initially, is to mark these fields as transient instead which gives the benefit of keeping the signature of the add method simple and interpretable. The downside is that it requires the use of `transient lazy val`s which are difficult to reason about if one is not quite familiar with serialization in Scala/Spark.
## How was this patch tested?
**MLlib**
**ML without patch**
**ML with patch**
Author: sethah <seth.hendrickson16@gmail.com>
Closes #14109 from sethah/LIR_serialize.
Diffstat (limited to 'docs/streaming-custom-receivers.md')
0 files changed, 0 insertions, 0 deletions