aboutsummaryrefslogtreecommitdiff
path: root/sql/core
diff options
context:
space:
mode:
authoryongtang <yongtang@users.noreply.github.com>2015-04-29 23:55:51 +0100
committerSean Owen <sowen@cloudera.com>2015-04-29 23:55:51 +0100
commit3fc6cfd079d8cdd35574605cb9a4178ca7f2613d (patch)
tree69d0f60f320df587d31d325aeed7a0db538acd64 /sql/core
parent7f4b583733714bbecb43fb0823134bf2ec720a17 (diff)
downloadspark-3fc6cfd079d8cdd35574605cb9a4178ca7f2613d.tar.gz
spark-3fc6cfd079d8cdd35574605cb9a4178ca7f2613d.tar.bz2
spark-3fc6cfd079d8cdd35574605cb9a4178ca7f2613d.zip
[SPARK-7155] [CORE] Allow newAPIHadoopFile to support comma-separated list of files as input
See JIRA: https://issues.apache.org/jira/browse/SPARK-7155 SparkContext's newAPIHadoopFile() does not support comma-separated list of files. For example, the following: ```scala sc.newAPIHadoopFile("/root/file1.txt,/root/file2.txt", classOf[TextInputFormat], classOf[LongWritable], classOf[Text]) ``` will throw ``` org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/root/file1.txt,/root/file2.txt ``` However, the other API hadoopFile() is able to process comma-separated list of files correctly. In addition, since sc.textFile() uses hadoopFile(), it is also able to process comma-separated list of files correctly. That means the behaviors of hadoopFile() and newAPIHadoopFile() are not aligned. This pull request fix this issue and allows newAPIHadoopFile() to support comma-separated list of files as input. A unit test has also been added in SparkContextSuite.scala. It creates two temporary text files as the input and tested against sc.textFile(), sc.hadoopFile(), and sc.newAPIHadoopFile(). Note: The contribution is my original work and that I license the work to the project under the project's open source license. Author: yongtang <yongtang@users.noreply.github.com> Closes #5708 from yongtang/SPARK-7155 and squashes the following commits: 654c80c [yongtang] [SPARK-7155] [CORE] Remove unneeded temp file deletion in unit test as parent dir is already temporary. 26faa6a [yongtang] [SPARK-7155] [CORE] Support comma-separated list of files as input for newAPIHadoopFile, wholeTextFiles, and binaryFiles. Use setInputPaths for consistency. 73e1f16 [yongtang] [SPARK-7155] [CORE] Allow newAPIHadoopFile to support comma-separated list of files as input.
Diffstat (limited to 'sql/core')
0 files changed, 0 insertions, 0 deletions