diff options
| author | Davies Liu <davies@databricks.com> | 2016-08-29 11:23:53 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-08-29 11:23:53 -0700 |
| commit | 48caec2516ef35bfa1a3de2dc0a80d0dc819e6bd (patch) | |
| tree | 2da91b214f23f1f559316e32e971dd6c03f70c47 /R | |
| parent | 6a0fda2c0590b455e8713da79cd5f2413e5d0f28 (diff) | |
| download | spark-48caec2516ef35bfa1a3de2dc0a80d0dc819e6bd.tar.gz spark-48caec2516ef35bfa1a3de2dc0a80d0dc819e6bd.tar.bz2 spark-48caec2516ef35bfa1a3de2dc0a80d0dc819e6bd.zip | |
[SPARK-17063] [SQL] Improve performance of MSCK REPAIR TABLE with Hive metastore
## What changes were proposed in this pull request?
This PR split the the single `createPartitions()` call into smaller batches, which could prevent Hive metastore from OOM (caused by millions of partitions).
It will also try to gather all the fast stats (number of files and total size of all files) in parallel to avoid the bottle neck of listing the files in metastore sequential, which is controlled by spark.sql.gatherFastStats (enabled by default).
## How was this patch tested?
Tested locally with 10000 partitions and 100 files with embedded metastore, without gathering fast stats in parallel, adding partitions took 153 seconds, after enable that, gathering the fast stats took about 34 seconds, adding these partitions took 25 seconds (most of the time spent in object store), 59 seconds in total, 2.5X faster (with larger cluster, gathering will much faster).
Author: Davies Liu <davies@databricks.com>
Closes #14607 from davies/repair_batch.
Diffstat (limited to 'R')
0 files changed, 0 insertions, 0 deletions