diff options
author | Liwei Lin <lwlin7@gmail.com> | 2016-04-26 23:08:40 -0700 |
---|---|---|
committer | Davies Liu <davies.liu@gmail.com> | 2016-04-26 23:08:40 -0700 |
commit | b2a45606481a6da6e1f68d14d1095a8dcf2a0e57 (patch) | |
tree | 8dd851542e31316beeb564ca9cc16de2fc4f8c80 | |
parent | 8fda5a73dc165fda2229a27c5a9e148b43b91c3a (diff) | |
download | spark-b2a45606481a6da6e1f68d14d1095a8dcf2a0e57.tar.gz spark-b2a45606481a6da6e1f68d14d1095a8dcf2a0e57.tar.bz2 spark-b2a45606481a6da6e1f68d14d1095a8dcf2a0e57.zip |
[SPARK-14911] [CORE] Fix a potential data race in TaskMemoryManager
## What changes were proposed in this pull request?
[[SPARK-13210][SQL] catch OOM when allocate memory and expand array](https://github.com/apache/spark/commit/37bc203c8dd5022cb11d53b697c28a737ee85bcc) introduced an `acquiredButNotUsed` field, but it might not be correctly synchronized:
- the write `acquiredButNotUsed += acquired` is guarded by `this` lock (see [here](https://github.com/apache/spark/blame/master/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java#L271));
- the read `memoryManager.releaseExecutionMemory(acquiredButNotUsed, taskAttemptId, tungstenMemoryMode)` (see [here](https://github.com/apache/spark/blame/master/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java#L400)) might not be correctly synchronized, and thus might not see `acquiredButNotUsed`'s most recent value.
This patch makes `acquiredButNotUsed` volatile to fix this.
## How was this patch tested?
This should be covered by existing suits.
Author: Liwei Lin <lwlin7@gmail.com>
Closes #12681 from lw-lin/fix-acquiredButNotUsed.
-rw-r--r-- | core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java b/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java index 6b7d9aa20f..2796114fc5 100644 --- a/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java +++ b/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java @@ -114,7 +114,7 @@ public class TaskMemoryManager { /** * The amount of memory that is acquired but not used. */ - private long acquiredButNotUsed = 0L; + private volatile long acquiredButNotUsed = 0L; /** * Construct a new TaskMemoryManager. |