diff options
| author | Pei-Lun Lee <pllee@appier.com> | 2015-04-28 16:50:18 +0800 |
|---|---|---|
| committer | Cheng Lian <lian@databricks.com> | 2015-04-28 16:50:18 +0800 |
| commit | e13cd86567a43672297bb488088dd8f40ec799bf (patch) | |
| tree | e5fe12313db2a082129d28d48b5bc56d656d98c9 /assembly | |
| parent | d94cd1a733d5715792e6c4eac87f0d5c81aebbe2 (diff) | |
| download | spark-e13cd86567a43672297bb488088dd8f40ec799bf.tar.gz spark-e13cd86567a43672297bb488088dd8f40ec799bf.tar.bz2 spark-e13cd86567a43672297bb488088dd8f40ec799bf.zip | |
[SPARK-6352] [SQL] Custom parquet output committer
Add new config "spark.sql.parquet.output.committer.class" to allow custom parquet output committer and an output committer class specific to use on s3.
Fix compilation error introduced by https://github.com/apache/spark/pull/5042.
Respect ParquetOutputFormat.ENABLE_JOB_SUMMARY flag.
Author: Pei-Lun Lee <pllee@appier.com>
Closes #5525 from ypcat/spark-6352 and squashes the following commits:
54c6b15 [Pei-Lun Lee] error handling
472870e [Pei-Lun Lee] add back custom parquet output committer
ddd0f69 [Pei-Lun Lee] Merge branch 'master' of https://github.com/apache/spark into spark-6352
9ece5c5 [Pei-Lun Lee] compatibility with hadoop 1.x
8413fcd [Pei-Lun Lee] Merge branch 'master' of https://github.com/apache/spark into spark-6352
fe65915 [Pei-Lun Lee] add support for parquet config parquet.enable.summary-metadata
e17bf47 [Pei-Lun Lee] Merge branch 'master' of https://github.com/apache/spark into spark-6352
9ae7545 [Pei-Lun Lee] [SPARL-6352] [SQL] Change to allow custom parquet output committer.
0d540b9 [Pei-Lun Lee] [SPARK-6352] [SQL] add license
c42468c [Pei-Lun Lee] [SPARK-6352] [SQL] add test case
0fc03ca [Pei-Lun Lee] [SPARK-6532] [SQL] hide class DirectParquetOutputCommitter
769bd67 [Pei-Lun Lee] DirectParquetOutputCommitter
f75e261 [Pei-Lun Lee] DirectParquetOutputCommitter
Diffstat (limited to 'assembly')
0 files changed, 0 insertions, 0 deletions