diff options
| author | Lianhui Wang <lianhuiwang09@gmail.com> | 2016-09-01 17:08:33 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2016-09-01 17:09:23 -0700 |
| commit | 06e33985c631fe91e1c4cef6039b8752548cc435 (patch) | |
| tree | e0096325b75f54a9c8e0397382e503c27c6a79bb /common | |
| parent | 5bea8757cce0b5e7c1f1ab9cd767c76fc63e2978 (diff) | |
| download | spark-06e33985c631fe91e1c4cef6039b8752548cc435.tar.gz spark-06e33985c631fe91e1c4cef6039b8752548cc435.tar.bz2 spark-06e33985c631fe91e1c4cef6039b8752548cc435.zip | |
[SPARK-16302][SQL] Set the right number of partitions for reading data from a local collection.
follow #13137 This pr sets the right number of partitions when reading data from a local collection.
Query 'val df = Seq((1, 2)).toDF("key", "value").count' always use defaultParallelism tasks. So it causes run many empty or small tasks.
Manually tested and checked.
Author: Lianhui Wang <lianhuiwang09@gmail.com>
Closes #13979 from lianhuiwang/localTable-Parallel.
Diffstat (limited to 'common')
0 files changed, 0 insertions, 0 deletions