diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2016-04-27 17:34:55 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-04-27 17:34:55 -0700 |
| commit | 8c49cebce572330fc84362662a9e3e8f7625bf5d (patch) | |
| tree | 9630ea3f1504f7be9655663ca23c6a54f12b160f /.gitignore | |
| parent | f5ebb18c45ffdee2756a80f64239cb9158df1a11 (diff) | |
| download | spark-8c49cebce572330fc84362662a9e3e8f7625bf5d.tar.gz spark-8c49cebce572330fc84362662a9e3e8f7625bf5d.tar.bz2 spark-8c49cebce572330fc84362662a9e3e8f7625bf5d.zip | |
[SPARK-14966] SizeEstimator should ignore classes in the scala.reflect package
In local profiling, I noticed SizeEstimator spending tons of time estimating the size of objects which contain TypeTag or ClassTag fields. The problem with these tags is that they reference global Scala reflection objects, which, in turn, reference many singletons, such as TestHive. This throws off the accuracy of the size estimation and wastes tons of time traversing a huge object graph.
As a result, I think that SizeEstimator should ignore any classes in the `scala.reflect` package.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #12741 from JoshRosen/ignore-scala-reflect-in-size-estimator.
Diffstat (limited to '.gitignore')
0 files changed, 0 insertions, 0 deletions