diff options
| author | Davies Liu <davies@databricks.com> | 2015-02-16 17:57:14 -0800 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-02-16 17:57:14 -0800 |
| commit | b1bd1dd3228ef50fa7310d466afd834b8cb1f22e (patch) | |
| tree | 681260e4b57602fceb7338efd113242e0fa0da7f /core | |
| parent | 1294a6e01af0d4f6678ea8cb5d47dc97112608b5 (diff) | |
| download | spark-b1bd1dd3228ef50fa7310d466afd834b8cb1f22e.tar.gz spark-b1bd1dd3228ef50fa7310d466afd834b8cb1f22e.tar.bz2 spark-b1bd1dd3228ef50fa7310d466afd834b8cb1f22e.zip | |
[SPARK-5788] [PySpark] capture the exception in python write thread
The exception in Python writer thread will shutdown executor.
Author: Davies Liu <davies@databricks.com>
Closes #4577 from davies/exception and squashes the following commits:
eb0ceff [Davies Liu] Update PythonRDD.scala
139b0db [Davies Liu] capture the exception in python write thread
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala b/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala index b89effc16d..2527211929 100644 --- a/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala +++ b/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala @@ -248,13 +248,13 @@ private[spark] class PythonRDD( } catch { case e: Exception if context.isCompleted || context.isInterrupted => logDebug("Exception thrown after task completion (likely due to cleanup)", e) - worker.shutdownOutput() + Utils.tryLog(worker.shutdownOutput()) case e: Exception => // We must avoid throwing exceptions here, because the thread uncaught exception handler // will kill the whole executor (see org.apache.spark.executor.Executor). _exception = e - worker.shutdownOutput() + Utils.tryLog(worker.shutdownOutput()) } finally { // Release memory used by this thread for shuffles env.shuffleMemoryManager.releaseMemoryForThisThread() |