diff options
| author | jerryshao <saisai.shao@intel.com> | 2014-08-01 04:32:46 -0700 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2014-08-01 04:32:46 -0700 |
| commit | a32f0fb73a739c56208cafcd9f08618fb6dd8859 (patch) | |
| tree | 30d2b0faec80084693f798047b3dc5e89971b31f /project/MimaExcludes.scala | |
| parent | 284771efbef2d6b22212afd49dd62732a2cf52a8 (diff) | |
| download | spark-a32f0fb73a739c56208cafcd9f08618fb6dd8859.tar.gz spark-a32f0fb73a739c56208cafcd9f08618fb6dd8859.tar.bz2 spark-a32f0fb73a739c56208cafcd9f08618fb6dd8859.zip | |
[SPARK-2103][Streaming] Change to ClassTag for KafkaInputDStream and fix reflection issue
This PR updates previous Manifest for KafkaInputDStream's Decoder to ClassTag, also fix the problem addressed in [SPARK-2103](https://issues.apache.org/jira/browse/SPARK-2103).
Previous Java interface cannot actually get the type of Decoder, so when using this Manifest to reconstruct the decode object will meet reflection exception.
Also for other two Java interfaces, ClassTag[String] is useless because calling Scala API will get the right implicit ClassTag.
Current Kafka unit test cannot actually verify the interface. I've tested these interfaces in my local and distribute settings.
Author: jerryshao <saisai.shao@intel.com>
Closes #1508 from jerryshao/SPARK-2103 and squashes the following commits:
e90c37b [jerryshao] Add Mima excludes
7529810 [jerryshao] Change Manifest to ClassTag for KafkaInputDStream's Decoder and fix Decoder construct issue when using Java API
Diffstat (limited to 'project/MimaExcludes.scala')
| -rw-r--r-- | project/MimaExcludes.scala | 7 |
1 files changed, 6 insertions, 1 deletions
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala index 5a835f5820..537ca0dcf2 100644 --- a/project/MimaExcludes.scala +++ b/project/MimaExcludes.scala @@ -71,7 +71,12 @@ object MimaExcludes { "org.apache.spark.storage.TachyonStore.putValues") ) ++ Seq( - ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.streaming.flume.FlumeReceiver.this") + ProblemFilters.exclude[MissingMethodProblem]( + "org.apache.spark.streaming.flume.FlumeReceiver.this"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]( + "org.apache.spark.streaming.kafka.KafkaUtils.createStream"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]( + "org.apache.spark.streaming.kafka.KafkaReceiver.this") ) ++ Seq( // Ignore some private methods in ALS. ProblemFilters.exclude[MissingMethodProblem]( |