diff options
| author | Michael Armbrust <michael@databricks.com> | 2014-12-03 14:13:35 -0800 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2014-12-03 14:13:35 -0800 |
| commit | 513ef82e85661552e596d0b483b645ac24e86d4d (patch) | |
| tree | ee1a169705a96ee41d340ea2e4de2bb475bbb34a /sql/core/src | |
| parent | 90ec643e9af4c8bbb9000edca08c07afb17939c7 (diff) | |
| download | spark-513ef82e85661552e596d0b483b645ac24e86d4d.tar.gz spark-513ef82e85661552e596d0b483b645ac24e86d4d.tar.bz2 spark-513ef82e85661552e596d0b483b645ac24e86d4d.zip | |
[SPARK-4552][SQL] Avoid exception when reading empty parquet data through Hive
This is a very small fix that catches one specific exception and returns an empty table. #3441 will address this in a more principled way.
Author: Michael Armbrust <michael@databricks.com>
Closes #3586 from marmbrus/fixEmptyParquet and squashes the following commits:
2781d9f [Michael Armbrust] Handle empty lists for newParquet
04dd376 [Michael Armbrust] Avoid exception when reading empty parquet data through Hive
Diffstat (limited to 'sql/core/src')
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala | 5 |
1 files changed, 4 insertions, 1 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala b/sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala index 9b89c3bfb3..14f8659f15 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala @@ -191,7 +191,10 @@ case class ParquetRelation2(path: String)(@transient val sqlContext: SQLContext) val selectedPartitions = partitions.filter(p => partitionFilters.forall(_(p))) val fs = FileSystem.get(new java.net.URI(path), sparkContext.hadoopConfiguration) val selectedFiles = selectedPartitions.flatMap(_.files).map(f => fs.makeQualified(f.getPath)) - org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, selectedFiles:_*) + // FileInputFormat cannot handle empty lists. + if (selectedFiles.nonEmpty) { + org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, selectedFiles: _*) + } // Push down filters when possible predicates |