diff options
| author | Sandy Ryza <sandy@cloudera.com> | 2015-05-05 12:38:46 +0100 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2015-05-05 12:38:54 +0100 |
| commit | 0327ca2b2c6ffb4dbd2770861a3b3ba806d55e58 (patch) | |
| tree | bb75726c20f7705622d1f7140fc79e73c1fb72e9 /.rat-excludes | |
| parent | 93af96a2f598f73b85285e91d09f939c0d6c1eb6 (diff) | |
| download | spark-0327ca2b2c6ffb4dbd2770861a3b3ba806d55e58.tar.gz spark-0327ca2b2c6ffb4dbd2770861a3b3ba806d55e58.tar.bz2 spark-0327ca2b2c6ffb4dbd2770861a3b3ba806d55e58.zip | |
[SPARK-5112] Expose SizeEstimator as a developer api
"The best way to size the amount of memory consumption your dataset will require is to create an RDD, put it into cache, and look at the SparkContext logs on your driver program. The logs will tell you how much memory each partition is consuming, which you can aggregate to get the total size of the RDD."
-the Tuning Spark page
This is a pain. It would be much nicer to expose simply functionality for understanding the memory footprint of a Java object.
Author: Sandy Ryza <sandy@cloudera.com>
Closes #3913 from sryza/sandy-spark-5112 and squashes the following commits:
8d9e082 [Sandy Ryza] Add SizeEstimator in org.apache.spark
2e1a906 [Sandy Ryza] Revert "Move SizeEstimator out of util"
93f4cd0 [Sandy Ryza] Move SizeEstimator out of util
e21c1f4 [Sandy Ryza] Remove unused import
798ab88 [Sandy Ryza] Update documentation and add to SparkContext
34c523c [Sandy Ryza] SPARK-5112. Expose SizeEstimator as a developer api
(cherry picked from commit 4222da68dc5360b7a2a8b8bdce231e887ac2f044)
Signed-off-by: Sean Owen <sowen@cloudera.com>
Diffstat (limited to '.rat-excludes')
0 files changed, 0 insertions, 0 deletions