diff options
| author | Andre Schumacher <andre.schumacher@iki.fi> | 2014-04-03 15:31:47 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@apache.org> | 2014-04-03 15:31:47 -0700 |
| commit | fbebaedf26286ee8a75065822a3af1148351f828 (patch) | |
| tree | d1c02f6c81b325e52b0ac08b7f55f12b670087e8 /sql/core/src/test/resources | |
| parent | 92a86b285f8a4af1bdf577dd4c4ea0fd5ca8d682 (diff) | |
| download | spark-fbebaedf26286ee8a75065822a3af1148351f828.tar.gz spark-fbebaedf26286ee8a75065822a3af1148351f828.tar.bz2 spark-fbebaedf26286ee8a75065822a3af1148351f828.zip | |
Spark parquet improvements
A few improvements to the Parquet support for SQL queries:
- Instead of files a ParquetRelation is now backed by a directory, which simplifies importing data from other
sources
- InsertIntoParquetTable operation now supports switching between overwriting or appending (at least in
HiveQL)
- tests now use the new API
- Parquet logging can be set to WARNING level (Default)
- Default compression for Parquet files (GZIP, as in parquet-mr)
Author: Andre Schumacher <andre.schumacher@iki.fi>
Closes #195 from AndreSchumacher/spark_parquet_improvements and squashes the following commits:
54df314 [Andre Schumacher] SPARK-1383 [SQL] Improvements to ParquetRelation
Diffstat (limited to 'sql/core/src/test/resources')
| -rw-r--r-- | sql/core/src/test/resources/log4j.properties | 8 |
1 files changed, 3 insertions, 5 deletions
diff --git a/sql/core/src/test/resources/log4j.properties b/sql/core/src/test/resources/log4j.properties index 7bb6789bd3..dffd15a618 100644 --- a/sql/core/src/test/resources/log4j.properties +++ b/sql/core/src/test/resources/log4j.properties @@ -45,8 +45,6 @@ log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=OFF log4j.additivity.hive.ql.metadata.Hive=false log4j.logger.hive.ql.metadata.Hive=OFF -# Parquet logging -parquet.hadoop.InternalParquetRecordReader=WARN -log4j.logger.parquet.hadoop.InternalParquetRecordReader=WARN -parquet.hadoop.ParquetInputFormat=WARN -log4j.logger.parquet.hadoop.ParquetInputFormat=WARN +# Parquet related logging +log4j.logger.parquet.hadoop=WARN +log4j.logger.org.apache.spark.sql.parquet=INFO |