diff options
| author | genmao.ygm <genmao.ygm@genmaoygmdeMacBook-Air.local> | 2016-11-15 10:32:43 -0800 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2016-11-15 10:32:43 -0800 |
| commit | 745ab8bc50da89c42b297de9dcb833e5f2074481 (patch) | |
| tree | cb79cc6fec8cf14bc4cdd6f510fefbd972748ec3 /streaming/src | |
| parent | f14ae4900ad0ed66ba36108b7792d56cd6767a69 (diff) | |
| download | spark-745ab8bc50da89c42b297de9dcb833e5f2074481.tar.gz spark-745ab8bc50da89c42b297de9dcb833e5f2074481.tar.bz2 spark-745ab8bc50da89c42b297de9dcb833e5f2074481.zip | |
[SPARK-18379][SQL] Make the parallelism of parallelPartitionDiscovery configurable.
## What changes were proposed in this pull request?
The largest parallelism in PartitioningAwareFileIndex #listLeafFilesInParallel() is 10000 in hard code. We may need to make this number configurable. And in PR, I reduce it to 100.
## How was this patch tested?
Existing ut.
Author: genmao.ygm <genmao.ygm@genmaoygmdeMacBook-Air.local>
Author: dylon <hustyugm@gmail.com>
Closes #15829 from uncleGen/SPARK-18379.
Diffstat (limited to 'streaming/src')
0 files changed, 0 insertions, 0 deletions