diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2015-03-24 14:38:20 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-03-24 14:38:20 -0700 |
| commit | 7215aa745590a3eec9c1ff35d28194235a550db7 (patch) | |
| tree | d4f59accb3689715013848452a7d19958e3c5e52 /repl/pom.xml | |
| parent | a8f51b82968147abebbe61b8b68b066d21a0c6e6 (diff) | |
| download | spark-7215aa745590a3eec9c1ff35d28194235a550db7.tar.gz spark-7215aa745590a3eec9c1ff35d28194235a550db7.tar.bz2 spark-7215aa745590a3eec9c1ff35d28194235a550db7.zip | |
[SPARK-6209] Clean up connections in ExecutorClassLoader after failing to load classes (master branch PR)
ExecutorClassLoader does not ensure proper cleanup of network connections that it opens. If it fails to load a class, it may leak partially-consumed InputStreams that are connected to the REPL's HTTP class server, causing that server to exhaust its thread pool, which can cause the entire job to hang. See [SPARK-6209](https://issues.apache.org/jira/browse/SPARK-6209) for more details, including a bug reproduction.
This patch fixes this issue by ensuring proper cleanup of these resources. It also adds logging for unexpected error cases.
This PR is an extended version of #4935 and adds a regression test.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #4944 from JoshRosen/executorclassloader-leak-master-branch and squashes the following commits:
e0e3c25 [Josh Rosen] Wrap try block around getReponseCode; re-enable keep-alive by closing error stream
961c284 [Josh Rosen] Roll back changes that were added to get the regression test to fail
7ee2261 [Josh Rosen] Add a failing regression test
e2d70a3 [Josh Rosen] Properly clean up after errors in ExecutorClassLoader
Diffstat (limited to 'repl/pom.xml')
| -rw-r--r-- | repl/pom.xml | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/repl/pom.xml b/repl/pom.xml index edfa1c7f2c..03053b4c3b 100644 --- a/repl/pom.xml +++ b/repl/pom.xml @@ -84,6 +84,11 @@ <artifactId>scalacheck_${scala.binary.version}</artifactId> <scope>test</scope> </dependency> + <dependency> + <groupId>org.mockito</groupId> + <artifactId>mockito-all</artifactId> + <scope>test</scope> + </dependency> <!-- Explicit listing of transitive deps that are shaded. Otherwise, odd compiler crashes. --> <dependency> |