diff options
| author | Sebastien Rainville <sebastien@hopper.com> | 2016-05-04 14:32:36 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2016-05-04 14:32:36 -0700 |
| commit | eb019af9a9cadb127eab1b6d30312169ed90f808 (patch) | |
| tree | 1fb16c735594776ff511304160c705b9fd7cee97 /repl | |
| parent | cdce4e62a5674e2034e5d395578b1a60e3d8c435 (diff) | |
| download | spark-eb019af9a9cadb127eab1b6d30312169ed90f808.tar.gz spark-eb019af9a9cadb127eab1b6d30312169ed90f808.tar.bz2 spark-eb019af9a9cadb127eab1b6d30312169ed90f808.zip | |
[SPARK-13001][CORE][MESOS] Prevent getting offers when reached max cores
Similar to https://github.com/apache/spark/pull/8639
This change rejects offers for 120s when reached `spark.cores.max` in coarse-grained mode to mitigate offer starvation. This prevents Mesos to send us offers again and again, starving other frameworks. This is especially problematic when running many small frameworks on the same Mesos cluster, e.g. many small Sparks streaming jobs, and cause the bigger spark jobs to stop receiving offers. By rejecting the offers for a long period of time, they become available to those other frameworks.
Author: Sebastien Rainville <sebastien@hopper.com>
Closes #10924 from sebastienrainville/master.
Diffstat (limited to 'repl')
0 files changed, 0 insertions, 0 deletions