diff options
| author | Yin Huai <yhuai@databricks.com> | 2015-11-18 00:42:52 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-11-18 00:42:52 -0800 |
| commit | 1714350bddd78cd1398e1a816f675ab729001081 (patch) | |
| tree | f567319f64994f8492c8d69c8aacea83d87b799a /core/src/main/java | |
| parent | 5e2b44474c2b838bebeffe5ba5cd72961b0cd31e (diff) | |
| download | spark-1714350bddd78cd1398e1a816f675ab729001081.tar.gz spark-1714350bddd78cd1398e1a816f675ab729001081.tar.bz2 spark-1714350bddd78cd1398e1a816f675ab729001081.zip | |
[SPARK-11792][SQL] SizeEstimator cannot provide a good size estimation of UnsafeHashedRelations
https://issues.apache.org/jira/browse/SPARK-11792
Right now, SizeEstimator will "think" a small UnsafeHashedRelation is several GBs.
Author: Yin Huai <yhuai@databricks.com>
Closes #9788 from yhuai/SPARK-11792.
Diffstat (limited to 'core/src/main/java')
| -rw-r--r-- | core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java b/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java index 5f743b2885..d31eb449eb 100644 --- a/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java +++ b/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java @@ -215,6 +215,9 @@ public class TaskMemoryManager { logger.info( "{} bytes of memory were used by task {} but are not associated with specific consumers", memoryNotAccountedFor, taskAttemptId); + logger.info( + "{} bytes of memory are used for execution and {} bytes of memory are used for storage", + memoryManager.executionMemoryUsed(), memoryManager.storageMemoryUsed()); } } |