diff options
| author | Hari Shreedharan <hshreedharan@apache.org> | 2014-08-27 02:39:02 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2014-08-27 02:39:02 -0700 |
| commit | 6f671d04fa98f97fd48c5e749b9f47dd4a8b4f44 (patch) | |
| tree | c56c24a614050b66506baf79d964ba6627117669 /external/flume-sink/pom.xml | |
| parent | 171a41cb034f4ea80f6a3c91a6872970de16a14a (diff) | |
| download | spark-6f671d04fa98f97fd48c5e749b9f47dd4a8b4f44.tar.gz spark-6f671d04fa98f97fd48c5e749b9f47dd4a8b4f44.tar.bz2 spark-6f671d04fa98f97fd48c5e749b9f47dd4a8b4f44.zip | |
[SPARK-3154][STREAMING] Make FlumePollingInputDStream shutdown cleaner.
Currently lot of errors get thrown from Avro IPC layer when the dstream
or sink is shutdown. This PR cleans it up. Some refactoring is done in the
receiver code to put all of the RPC code into a single Try and just recover
from that. The sink code has also been cleaned up.
Author: Hari Shreedharan <hshreedharan@apache.org>
Closes #2065 from harishreedharan/clean-flume-shutdown and squashes the following commits:
f93a07c [Hari Shreedharan] Formatting fixes.
d7427cc [Hari Shreedharan] More fixes!
a0a8852 [Hari Shreedharan] Fix race condition, hopefully! Minor other changes.
4c9ed02 [Hari Shreedharan] Remove unneeded list in Callback handler. Other misc changes.
8fee36f [Hari Shreedharan] Scala-library is required, else maven build fails. Also catch InterruptedException in TxnProcessor.
445e700 [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into clean-flume-shutdown
87232e0 [Hari Shreedharan] Refactor Flume Input Stream. Clean up code, better error handling.
9001d26 [Hari Shreedharan] Change log level to debug in TransactionProcessor#shutdown method
e7b8d82 [Hari Shreedharan] Incorporate review feedback
598efa7 [Hari Shreedharan] Clean up some exception handling code
e1027c6 [Hari Shreedharan] Merge remote-tracking branch 'asf/master' into clean-flume-shutdown
ed608c8 [Hari Shreedharan] [SPARK-3154][STREAMING] Make FlumePollingInputDStream shutdown cleaner.
Diffstat (limited to 'external/flume-sink/pom.xml')
| -rw-r--r-- | external/flume-sink/pom.xml | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index c1e8e65464..b345276b08 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -71,6 +71,10 @@ <scope>test</scope> </dependency> <dependency> + <groupId>org.scala-lang</groupId> + <artifactId>scala-library</artifactId> + </dependency> + <dependency> <!-- Netty explicitly added in test as it has been excluded from Flume dependency (to avoid runtime problems when running with |