diff options
| author | Cheng Lian <lian@databricks.com> | 2016-03-30 18:21:06 -0700 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2016-03-30 18:21:06 -0700 |
| commit | 26445c2e472bad137fd350e4089dd0ff43a42039 (patch) | |
| tree | 7972c24c16fef4202224d9982edb6698ece7e589 /sql/core/src/test/scala | |
| parent | da54abfd8730ef752eca921089bcf568773bd24a (diff) | |
| download | spark-26445c2e472bad137fd350e4089dd0ff43a42039.tar.gz spark-26445c2e472bad137fd350e4089dd0ff43a42039.tar.bz2 spark-26445c2e472bad137fd350e4089dd0ff43a42039.zip | |
[SPARK-14206][SQL] buildReader() implementation for CSV
## What changes were proposed in this pull request?
Major changes:
1. Implement `FileFormat.buildReader()` for the CSV data source.
1. Add an extra argument to `FileFormat.buildReader()`, `physicalSchema`, which is basically the result of `FileFormat.inferSchema` or user specified schema.
This argument is necessary because the CSV data source needs to know all the columns of the underlying files to read the file.
## How was this patch tested?
Existing tests should do the work.
Author: Cheng Lian <lian@databricks.com>
Closes #12002 from liancheng/spark-14206-csv-build-reader.
Diffstat (limited to 'sql/core/src/test/scala')
| -rw-r--r-- | sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala index 45620bc965..717a3a80b7 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala @@ -376,14 +376,15 @@ class TestFileFormat extends FileFormat { override def buildReader( sqlContext: SQLContext, - partitionSchema: StructType, dataSchema: StructType, + partitionSchema: StructType, + requiredSchema: StructType, filters: Seq[Filter], options: Map[String, String]): PartitionedFile => Iterator[InternalRow] = { // Record the arguments so they can be checked in the test case. LastArguments.partitionSchema = partitionSchema - LastArguments.dataSchema = dataSchema + LastArguments.dataSchema = requiredSchema LastArguments.filters = filters LastArguments.options = options |