diff options
| author | Pei-Lun Lee <pllee@appier.com> | 2015-04-13 21:52:00 +0800 |
|---|---|---|
| committer | Cheng Lian <lian@databricks.com> | 2015-04-13 21:52:00 +0800 |
| commit | b29663eeea440b1d1a288d41b5ddf67e77c5bd54 (patch) | |
| tree | 66f09463331170773aea542ad90e4f6f8f06685e /core/src/test | |
| parent | 202ebf06e0f2d5df8b712e604fd95fa58e34ea20 (diff) | |
| download | spark-b29663eeea440b1d1a288d41b5ddf67e77c5bd54.tar.gz spark-b29663eeea440b1d1a288d41b5ddf67e77c5bd54.tar.bz2 spark-b29663eeea440b1d1a288d41b5ddf67e77c5bd54.zip | |
[SPARK-6352] [SQL] Add DirectParquetOutputCommitter
Add a DirectParquetOutputCommitter class that skips _temporary directory when saving to s3. Add new config value "spark.sql.parquet.useDirectParquetOutputCommitter" (default false) to choose between the default output committer.
Author: Pei-Lun Lee <pllee@appier.com>
Closes #5042 from ypcat/spark-6352 and squashes the following commits:
e17bf47 [Pei-Lun Lee] Merge branch 'master' of https://github.com/apache/spark into spark-6352
9ae7545 [Pei-Lun Lee] [SPARL-6352] [SQL] Change to allow custom parquet output committer.
0d540b9 [Pei-Lun Lee] [SPARK-6352] [SQL] add license
c42468c [Pei-Lun Lee] [SPARK-6352] [SQL] add test case
0fc03ca [Pei-Lun Lee] [SPARK-6532] [SQL] hide class DirectParquetOutputCommitter
769bd67 [Pei-Lun Lee] DirectParquetOutputCommitter
f75e261 [Pei-Lun Lee] DirectParquetOutputCommitter
Diffstat (limited to 'core/src/test')
0 files changed, 0 insertions, 0 deletions