diff options
| author | Davies Liu <davies@databricks.com> | 2015-04-02 12:18:33 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-04-02 12:18:48 -0700 |
| commit | ee2bd70a48e35ab045a440328f97d88055b4dc28 (patch) | |
| tree | eabab92cfeafc521ef605132221868ebbebd7991 | |
| parent | 1160cc9e1883fb4704c46e7983799671313f9f39 (diff) | |
| download | spark-ee2bd70a48e35ab045a440328f97d88055b4dc28.tar.gz spark-ee2bd70a48e35ab045a440328f97d88055b4dc28.tar.bz2 spark-ee2bd70a48e35ab045a440328f97d88055b4dc28.zip | |
[SPARK-6667] [PySpark] remove setReuseAddress
The reused address on server side had caused the server can not acknowledge the connected connections, remove it.
This PR will retry once after timeout, it also add a timeout at client side.
Author: Davies Liu <davies@databricks.com>
Closes #5324 from davies/collect_hang and squashes the following commits:
e5a51a2 [Davies Liu] remove setReuseAddress
7977c2f [Davies Liu] do retry on client side
b838f35 [Davies Liu] retry after timeout
(cherry picked from commit 0cce5451adfc6bf4661bcf67aca3db26376455fe)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
| -rw-r--r-- | core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala | 1 | ||||
| -rw-r--r-- | python/pyspark/rdd.py | 1 |
2 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala b/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala index 4c71b69069..eaff5e5812 100644 --- a/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala +++ b/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala @@ -604,7 +604,6 @@ private[spark] object PythonRDD extends Logging { */ private def serveIterator[T](items: Iterator[T], threadName: String): Int = { val serverSocket = new ServerSocket(0, 1) - serverSocket.setReuseAddress(true) // Close the socket if no connection in 3 seconds serverSocket.setSoTimeout(3000) diff --git a/python/pyspark/rdd.py b/python/pyspark/rdd.py index bf17f513c0..a1974de261 100644 --- a/python/pyspark/rdd.py +++ b/python/pyspark/rdd.py @@ -113,6 +113,7 @@ def _parse_memory(s): def _load_from_socket(port, serializer): sock = socket.socket() + sock.settimeout(3) try: sock.connect(("localhost", port)) rf = sock.makefile("rb", 65536) |