diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2016-03-02 10:26:47 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2016-03-02 10:26:47 -0800 |
| commit | d6969ffc0f86c8a4ea0e94d06cb227178b000962 (patch) | |
| tree | 85c4daf02f27ad110d9cba1b713eff49c3f154e4 /core/src/main/scala/org/apache/spark/SparkEnv.scala | |
| parent | 8f8d8a2315514cd1f3609bc06e5cf6e6d06fdd91 (diff) | |
| download | spark-d6969ffc0f86c8a4ea0e94d06cb227178b000962.tar.gz spark-d6969ffc0f86c8a4ea0e94d06cb227178b000962.tar.bz2 spark-d6969ffc0f86c8a4ea0e94d06cb227178b000962.zip | |
[SPARK-12817] Add BlockManager.getOrElseUpdate and remove CacheManager
CacheManager directly calls MemoryStore.unrollSafely() and has its own logic for handling graceful fallback to disk when cached data does not fit in memory. However, this logic also exists inside of the MemoryStore itself, so this appears to be unnecessary duplication.
Thanks to the addition of block-level read/write locks in #10705, we can refactor the code to remove the CacheManager and replace it with an atomic `BlockManager.getOrElseUpdate()` method.
This pull request replaces / subsumes #10748.
/cc andrewor14 and nongli for review. Note that this changes the locking semantics of a couple of internal BlockManager methods (`doPut()` and `lockNewBlockForWriting`), so please pay attention to the Scaladoc changes and new test cases for those methods.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #11436 from JoshRosen/remove-cachemanager.
Diffstat (limited to 'core/src/main/scala/org/apache/spark/SparkEnv.scala')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/SparkEnv.scala | 4 |
1 files changed, 0 insertions, 4 deletions
diff --git a/core/src/main/scala/org/apache/spark/SparkEnv.scala b/core/src/main/scala/org/apache/spark/SparkEnv.scala index 204f7356f7..b3b3729625 100644 --- a/core/src/main/scala/org/apache/spark/SparkEnv.scala +++ b/core/src/main/scala/org/apache/spark/SparkEnv.scala @@ -56,7 +56,6 @@ class SparkEnv ( private[spark] val rpcEnv: RpcEnv, val serializer: Serializer, val closureSerializer: Serializer, - val cacheManager: CacheManager, val mapOutputTracker: MapOutputTracker, val shuffleManager: ShuffleManager, val broadcastManager: BroadcastManager, @@ -333,8 +332,6 @@ object SparkEnv extends Logging { val broadcastManager = new BroadcastManager(isDriver, conf, securityManager) - val cacheManager = new CacheManager(blockManager) - val metricsSystem = if (isDriver) { // Don't start metrics system right now for Driver. // We need to wait for the task scheduler to give us an app ID. @@ -371,7 +368,6 @@ object SparkEnv extends Logging { rpcEnv, serializer, closureSerializer, - cacheManager, mapOutputTracker, shuffleManager, broadcastManager, |