diff options
| author | Andrew Or <andrew@databricks.com> | 2016-05-11 12:58:57 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-05-11 12:58:57 -0700 |
| commit | bb88ad4e0e870c88d474c71939a19541522a3023 (patch) | |
| tree | 23d1307530e5b20ba9ed26f2de42aaf4629ae8cc /python/pyspark/sql/types.py | |
| parent | 81c68eceba3a857ba7349c6892dc336c3ebd11dc (diff) | |
| download | spark-bb88ad4e0e870c88d474c71939a19541522a3023.tar.gz spark-bb88ad4e0e870c88d474c71939a19541522a3023.tar.bz2 spark-bb88ad4e0e870c88d474c71939a19541522a3023.zip | |
[SPARK-15260] Atomically resize memory pools
## What changes were proposed in this pull request?
When we acquire execution memory, we do a lot of things between shrinking the storage memory pool and enlarging the execution memory pool. In particular, we call `memoryStore.evictBlocksToFreeSpace`, which may do a lot of I/O and can throw exceptions. If an exception is thrown, the pool sizes on that executor will be in a bad state.
This patch minimizes the things we do between the two calls to make the resizing more atomic.
## How was this patch tested?
Jenkins.
Author: Andrew Or <andrew@databricks.com>
Closes #13039 from andrewor14/safer-pool.
Diffstat (limited to 'python/pyspark/sql/types.py')
0 files changed, 0 insertions, 0 deletions