diff options
author | Davies Liu <davies.liu@gmail.com> | 2014-08-10 13:00:38 -0700 |
---|---|---|
committer | Josh Rosen <joshrosen@apache.org> | 2014-08-10 13:00:38 -0700 |
commit | 28dcbb531ae57dc50f15ad9df6c31022731669c9 (patch) | |
tree | 307bfd292e7c0d141631191ae721e8eac96eabfd /core | |
parent | 1d03a26a4895c24ebfab1a3cf6656af75cb53003 (diff) | |
download | spark-28dcbb531ae57dc50f15ad9df6c31022731669c9.tar.gz spark-28dcbb531ae57dc50f15ad9df6c31022731669c9.tar.bz2 spark-28dcbb531ae57dc50f15ad9df6c31022731669c9.zip |
[SPARK-2898] [PySpark] fix bugs in deamon.py
1. do not use signal handler for SIGCHILD, it's easy to cause deadlock
2. handle EINTR during accept()
3. pass errno into JVM
4. handle EAGAIN during fork()
Now, it can pass 50k tasks tests in 180 seconds.
Author: Davies Liu <davies.liu@gmail.com>
Closes #1842 from davies/qa and squashes the following commits:
f0ea451 [Davies Liu] fix lint
03a2e8c [Davies Liu] cleanup dead children every seconds
32cb829 [Davies Liu] fix lint
0cd0817 [Davies Liu] fix bugs in deamon.py
Diffstat (limited to 'core')
-rw-r--r-- | core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala b/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala index 7af260d0b7..bf716a8ab0 100644 --- a/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala +++ b/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala @@ -68,7 +68,7 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String val socket = new Socket(daemonHost, daemonPort) val pid = new DataInputStream(socket.getInputStream).readInt() if (pid < 0) { - throw new IllegalStateException("Python daemon failed to launch worker") + throw new IllegalStateException("Python daemon failed to launch worker with code " + pid) } daemonWorkers.put(socket, pid) socket |