diff options
| author | Cheng Lian <lian@databricks.com> | 2015-03-31 07:48:37 +0800 |
|---|---|---|
| committer | Cheng Lian <lian@databricks.com> | 2015-03-31 07:48:37 +0800 |
| commit | fde6945417355ae57500b67d034c9cad4f20d240 (patch) | |
| tree | 94b8c986ac5174412e5055b60dd6f0dea760d871 /python/pyspark/sql/dataframe.py | |
| parent | f76d2e55b1a67bf5576e1aa001a0b872b9b3895a (diff) | |
| download | spark-fde6945417355ae57500b67d034c9cad4f20d240.tar.gz spark-fde6945417355ae57500b67d034c9cad4f20d240.tar.bz2 spark-fde6945417355ae57500b67d034c9cad4f20d240.zip | |
[SPARK-6369] [SQL] Uses commit coordinator to help committing Hive and Parquet tables
This PR leverages the output commit coordinator introduced in #4066 to help committing Hive and Parquet tables.
This PR extracts output commit code in `SparkHadoopWriter.commit` to `SparkHadoopMapRedUtil.commitTask`, and reuses it for committing Parquet and Hive tables on executor side.
TODO
- [ ] Add tests
<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/5139)
<!-- Reviewable:end -->
Author: Cheng Lian <lian@databricks.com>
Closes #5139 from liancheng/spark-6369 and squashes the following commits:
72eb628 [Cheng Lian] Fixes typo in javadoc
9a4b82b [Cheng Lian] Adds javadoc and addresses @aarondav's comments
dfdf3ef [Cheng Lian] Uses commit coordinator to help committing Hive and Parquet tables
Diffstat (limited to 'python/pyspark/sql/dataframe.py')
0 files changed, 0 insertions, 0 deletions