diff options
| author | Aaron Davidson <aaron@databricks.com> | 2014-10-10 01:44:36 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@apache.org> | 2014-10-10 01:44:36 -0700 |
| commit | 90f73fcc47c7bf881f808653d46a9936f37c3c31 (patch) | |
| tree | b1be2d788a28d267bf4fa96eb467954de18aae63 /python/pyspark/serializers.py | |
| parent | 411cf29fff011561f0093bb6101af87842828369 (diff) | |
| download | spark-90f73fcc47c7bf881f808653d46a9936f37c3c31.tar.gz spark-90f73fcc47c7bf881f808653d46a9936f37c3c31.tar.bz2 spark-90f73fcc47c7bf881f808653d46a9936f37c3c31.zip | |
[SPARK-3889] Attempt to avoid SIGBUS by not mmapping files in ConnectionManager
In general, individual shuffle blocks are frequently small, so mmapping them often creates a lot of waste. It may not be bad to mmap the larger ones, but it is pretty inconvenient to get configuration into ManagedBuffer, and besides it is unlikely to help all that much.
Author: Aaron Davidson <aaron@databricks.com>
Closes #2742 from aarondav/mmap and squashes the following commits:
a152065 [Aaron Davidson] Add other pathway back
52b6cd2 [Aaron Davidson] [SPARK-3889] Attempt to avoid SIGBUS by not mmapping files in ConnectionManager
Diffstat (limited to 'python/pyspark/serializers.py')
0 files changed, 0 insertions, 0 deletions