diff options
| author | Aaron Davidson <aaron@databricks.com> | 2015-04-20 09:54:21 -0700 |
|---|---|---|
| committer | Aaron Davidson <aaron@databricks.com> | 2015-04-20 09:54:21 -0700 |
| commit | 968ad972175390bb0a96918fd3c7b318d70fa466 (patch) | |
| tree | 35b66ccd71daf19f9556da4afbf0e1fd60413ea8 /python/pyspark/sql/context.py | |
| parent | 1be207078cef48c5935595969bf9f6b1ec1334ca (diff) | |
| download | spark-968ad972175390bb0a96918fd3c7b318d70fa466.tar.gz spark-968ad972175390bb0a96918fd3c7b318d70fa466.tar.bz2 spark-968ad972175390bb0a96918fd3c7b318d70fa466.zip | |
[SPARK-7003] Improve reliability of connection failure detection between Netty block transfer service endpoints
Currently we rely on the assumption that an exception will be raised and the channel closed if two endpoints cannot communicate over a Netty TCP channel. However, this guarantee does not hold in all network environments, and [SPARK-6962](https://issues.apache.org/jira/browse/SPARK-6962) seems to point to a case where only the server side of the connection detected a fault.
This patch improves robustness of fetch/rpc requests by having an explicit timeout in the transport layer which closes the connection if there is a period of inactivity while there are outstanding requests.
NB: This patch is actually only around 50 lines added if you exclude the testing-related code.
Author: Aaron Davidson <aaron@databricks.com>
Closes #5584 from aarondav/timeout and squashes the following commits:
8699680 [Aaron Davidson] Address Reynold's comments
37ce656 [Aaron Davidson] [SPARK-7003] Improve reliability of connection failure detection between Netty block transfer service endpoints
Diffstat (limited to 'python/pyspark/sql/context.py')
0 files changed, 0 insertions, 0 deletions