diff options
author | yongtang <yongtang@users.noreply.github.com> | 2015-04-29 23:55:51 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2015-04-29 23:55:51 +0100 |
commit | 3fc6cfd079d8cdd35574605cb9a4178ca7f2613d (patch) | |
tree | 69d0f60f320df587d31d325aeed7a0db538acd64 /sql/core | |
parent | 7f4b583733714bbecb43fb0823134bf2ec720a17 (diff) | |
download | spark-3fc6cfd079d8cdd35574605cb9a4178ca7f2613d.tar.gz spark-3fc6cfd079d8cdd35574605cb9a4178ca7f2613d.tar.bz2 spark-3fc6cfd079d8cdd35574605cb9a4178ca7f2613d.zip |
[SPARK-7155] [CORE] Allow newAPIHadoopFile to support comma-separated list of files as input
See JIRA: https://issues.apache.org/jira/browse/SPARK-7155
SparkContext's newAPIHadoopFile() does not support comma-separated list of files. For example, the following:
```scala
sc.newAPIHadoopFile("/root/file1.txt,/root/file2.txt", classOf[TextInputFormat], classOf[LongWritable], classOf[Text])
```
will throw
```
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/root/file1.txt,/root/file2.txt
```
However, the other API hadoopFile() is able to process comma-separated list of files correctly. In addition, since sc.textFile() uses hadoopFile(), it is also able to process comma-separated list of files correctly.
That means the behaviors of hadoopFile() and newAPIHadoopFile() are not aligned.
This pull request fix this issue and allows newAPIHadoopFile() to support comma-separated list of files as input.
A unit test has also been added in SparkContextSuite.scala. It creates two temporary text files as the input and tested against sc.textFile(), sc.hadoopFile(), and sc.newAPIHadoopFile().
Note: The contribution is my original work and that I license the work to the project under the project's open source license.
Author: yongtang <yongtang@users.noreply.github.com>
Closes #5708 from yongtang/SPARK-7155 and squashes the following commits:
654c80c [yongtang] [SPARK-7155] [CORE] Remove unneeded temp file deletion in unit test as parent dir is already temporary.
26faa6a [yongtang] [SPARK-7155] [CORE] Support comma-separated list of files as input for newAPIHadoopFile, wholeTextFiles, and binaryFiles. Use setInputPaths for consistency.
73e1f16 [yongtang] [SPARK-7155] [CORE] Allow newAPIHadoopFile to support comma-separated list of files as input.
Diffstat (limited to 'sql/core')
0 files changed, 0 insertions, 0 deletions