diff options
author | Sean Owen <sowen@cloudera.com> | 2015-02-16 19:32:31 +0000 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2015-02-16 19:32:31 +0000 |
commit | 8e25373ce72061d3b6a353259ec627606afa4a5f (patch) | |
tree | 11ccef85d99cd31fb3bfd5d526eef0ce2629c3cd /streaming/src/test | |
parent | 9baac56ccd57d3890a9b6439d4e13bbe9381822b (diff) | |
download | spark-8e25373ce72061d3b6a353259ec627606afa4a5f.tar.gz spark-8e25373ce72061d3b6a353259ec627606afa4a5f.tar.bz2 spark-8e25373ce72061d3b6a353259ec627606afa4a5f.zip |
SPARK-5795 [STREAMING] api.java.JavaPairDStream.saveAsNewAPIHadoopFiles may not friendly to java
Revise JavaPairDStream API declaration on saveAs Hadoop methods, to allow it to be called directly as intended.
CC tdas for review
Author: Sean Owen <sowen@cloudera.com>
Closes #4608 from srowen/SPARK-5795 and squashes the following commits:
36f1ead [Sean Owen] Add code that shows compile problem and fix
036bd27 [Sean Owen] Revise JavaPairDStream API declaration on saveAs Hadoop methods, to allow it to be called directly as intended.
Diffstat (limited to 'streaming/src/test')
-rw-r--r-- | streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java | 18 |
1 files changed, 18 insertions, 0 deletions
diff --git a/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java b/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java index 2df8cf6a8a..57302ff407 100644 --- a/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java +++ b/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java @@ -1828,4 +1828,22 @@ public class JavaAPISuite extends LocalJavaStreamingContext implements Serializa return expected; } + + // SPARK-5795: no logic assertions, just testing that intended API invocations compile + private void compileSaveAsJavaAPI(JavaPairDStream<LongWritable,Text> pds) { + pds.saveAsNewAPIHadoopFiles( + "", "", LongWritable.class, Text.class, + org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.class); + pds.saveAsHadoopFiles( + "", "", LongWritable.class, Text.class, + org.apache.hadoop.mapred.SequenceFileOutputFormat.class); + // Checks that a previous common workaround for this API still compiles + pds.saveAsNewAPIHadoopFiles( + "", "", LongWritable.class, Text.class, + (Class) org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.class); + pds.saveAsHadoopFiles( + "", "", LongWritable.class, Text.class, + (Class) org.apache.hadoop.mapred.SequenceFileOutputFormat.class); + } + } |