diff options
| author | Akshat Aranya <aaranya@quantcast.com> | 2015-05-22 22:03:31 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2015-05-22 22:03:31 -0700 |
| commit | a16357413d2823bcc1d1bf55b4da191dc9b1b69a (patch) | |
| tree | 55fa8fa2bc4e33267a29d9962ec3af65d70567be /dev | |
| parent | 63a5ce75eac48a297751ac505d70ce4d47daf903 (diff) | |
| download | spark-a16357413d2823bcc1d1bf55b4da191dc9b1b69a.tar.gz spark-a16357413d2823bcc1d1bf55b4da191dc9b1b69a.tar.bz2 spark-a16357413d2823bcc1d1bf55b4da191dc9b1b69a.zip | |
[SPARK-7795] [CORE] Speed up task scheduling in standalone mode by reusing serializer
My experiments with scheduling very short tasks in standalone cluster mode indicated that a significant amount of time was being spent in scheduling the tasks (>500ms for 256 tasks). I found that most of the time was being spent in creating a new instance of serializer for each task. Changing this to just one serializer brought down the scheduling time to 8ms.
Author: Akshat Aranya <aaranya@quantcast.com>
Closes #6323 from coolfrood/master and squashes the following commits:
12d8c9e [Akshat Aranya] Reduce visibility of serializer
bd4a5dd [Akshat Aranya] Style fix
0b8ca93 [Akshat Aranya] Incorporate review comments
fe530cd [Akshat Aranya] Speed up task scheduling in standalone mode by reusing serializer instead of creating a new one for each task.
Diffstat (limited to 'dev')
0 files changed, 0 insertions, 0 deletions