diff options
| author | Cheng Lian <lian@databricks.com> | 2015-03-31 11:21:15 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2015-03-31 11:21:15 -0700 |
| commit | 81020144708773ba3af4932288ffa09ef901269e (patch) | |
| tree | 8eda2f98a4b70147804daa582abfb127b5fee093 /python/pyspark/mllib | |
| parent | a7992ffaf1e8adc9d2c225a986fa3162e8e130eb (diff) | |
| download | spark-81020144708773ba3af4932288ffa09ef901269e.tar.gz spark-81020144708773ba3af4932288ffa09ef901269e.tar.bz2 spark-81020144708773ba3af4932288ffa09ef901269e.zip | |
[SPARK-6575] [SQL] Adds configuration to disable schema merging while converting metastore Parquet tables
Consider a metastore Parquet table that
1. doesn't have schema evolution issue
2. has lots of data files and/or partitions
In this case, driver schema merging can be both slow and unnecessary. Would be good to have a configuration to let the use disable schema merging when converting such a metastore Parquet table.
<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/5231)
<!-- Reviewable:end -->
Author: Cheng Lian <lian@databricks.com>
Closes #5231 from liancheng/spark-6575 and squashes the following commits:
cd96159 [Cheng Lian] Adds configuration to disable schema merging while converting metastore Parquet tables
Diffstat (limited to 'python/pyspark/mllib')
0 files changed, 0 insertions, 0 deletions