diff options
| author | Nick Lavers <nick.lavers@videoamp.com> | 2017-01-17 12:14:38 +0000 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2017-01-17 12:14:38 +0000 |
| commit | 0019005a2d0f150fd00ad926d054a8beca4bbd68 (patch) | |
| tree | 5e51517dad00b0d4848206cce57c51774805933d | |
| parent | a774bca05ec6dd0deec638048dc8672a84427f49 (diff) | |
| download | spark-0019005a2d0f150fd00ad926d054a8beca4bbd68.tar.gz spark-0019005a2d0f150fd00ad926d054a8beca4bbd68.tar.bz2 spark-0019005a2d0f150fd00ad926d054a8beca4bbd68.zip | |
[SPARK-19219][SQL] Fix Parquet log output defaults
## What changes were proposed in this pull request?
Changing the default parquet logging levels to reflect the changes made in PR [#15538](https://github.com/apache/spark/pull/15538), in order to prevent the flood of log messages by default.
## How was this patch tested?
Default log output when reading from parquet 1.6 files was compared with and without this change. The change eliminates the extraneous logging and makes the output readable.
Author: Nick Lavers <nick.lavers@videoamp.com>
Closes #16580 from nicklavers/spark-19219-set_default_parquet_log_level.
| -rw-r--r-- | core/src/main/resources/org/apache/spark/log4j-defaults.properties | 4 | ||||
| -rw-r--r-- | sql/core/src/test/resources/log4j.properties | 4 | ||||
| -rw-r--r-- | sql/hive/src/test/resources/log4j.properties | 4 |
3 files changed, 8 insertions, 4 deletions
diff --git a/core/src/main/resources/org/apache/spark/log4j-defaults.properties b/core/src/main/resources/org/apache/spark/log4j-defaults.properties index 89a7963a86..2770100150 100644 --- a/core/src/main/resources/org/apache/spark/log4j-defaults.properties +++ b/core/src/main/resources/org/apache/spark/log4j-defaults.properties @@ -36,3 +36,7 @@ log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO # SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR + +# Parquet related logging +log4j.logger.org.apache.parquet.CorruptStatistics=ERROR +log4j.logger.parquet.CorruptStatistics=ERROR diff --git a/sql/core/src/test/resources/log4j.properties b/sql/core/src/test/resources/log4j.properties index 25b8173821..2e5cac1295 100644 --- a/sql/core/src/test/resources/log4j.properties +++ b/sql/core/src/test/resources/log4j.properties @@ -53,5 +53,5 @@ log4j.additivity.hive.ql.metadata.Hive=false log4j.logger.hive.ql.metadata.Hive=OFF # Parquet related logging -log4j.logger.org.apache.parquet=ERROR -log4j.logger.parquet=ERROR +log4j.logger.org.apache.parquet.CorruptStatistics=ERROR +log4j.logger.parquet.CorruptStatistics=ERROR diff --git a/sql/hive/src/test/resources/log4j.properties b/sql/hive/src/test/resources/log4j.properties index 072bb25d30..a48ae9fc5e 100644 --- a/sql/hive/src/test/resources/log4j.properties +++ b/sql/hive/src/test/resources/log4j.properties @@ -61,5 +61,5 @@ log4j.additivity.org.apache.hadoop.hive.ql.io.RCFile=false log4j.logger.org.apache.hadoop.hive.ql.io.RCFile=ERROR # Parquet related logging -log4j.logger.org.apache.parquet=ERROR -log4j.logger.parquet=ERROR +log4j.logger.org.apache.parquet.CorruptStatistics=ERROR +log4j.logger.parquet.CorruptStatistics=ERROR |