diff options
author | Andrew Or <andrew@databricks.com> | 2015-08-01 11:57:14 -0700 |
---|---|---|
committer | Andrew Or <andrew@databricks.com> | 2015-08-01 11:57:14 -0700 |
commit | 6688ba6e68e342201b81ea09cc2c6ba216f90f3e (patch) | |
tree | b5b951e88275e45705fb9466f1ea3bd4ec571818 /yarn | |
parent | c5166f7a69faeaa8a41a774c73c1ed4d4c2cf0ce (diff) | |
download | spark-6688ba6e68e342201b81ea09cc2c6ba216f90f3e.tar.gz spark-6688ba6e68e342201b81ea09cc2c6ba216f90f3e.tar.bz2 spark-6688ba6e68e342201b81ea09cc2c6ba216f90f3e.zip |
[SPARK-4751] Dynamic allocation in standalone mode
Dynamic allocation is a feature that allows a Spark application to scale the number of executors up and down dynamically based on the workload. Support was first introduced in YARN since 1.2, and then extended to Mesos coarse-grained mode recently. Today, it is finally supported in standalone mode as well!
I tested this locally and it works as expected. This is WIP because unit tests are coming.
Author: Andrew Or <andrew@databricks.com>
Closes #7532 from andrewor14/standalone-da and squashes the following commits:
b3c1736 [Andrew Or] Merge branch 'master' of github.com:apache/spark into standalone-da
879e928 [Andrew Or] Add end-to-end tests for standalone dynamic allocation
accc8f6 [Andrew Or] Address comments
ee686a8 [Andrew Or] Merge branch 'master' of github.com:apache/spark into standalone-da
c0a2c02 [Andrew Or] Fix build after merge conflict
24149eb [Andrew Or] Merge branch 'master' of github.com:apache/spark into standalone-da
2e762d6 [Andrew Or] Merge branch 'master' of github.com:apache/spark into standalone-da
6832bd7 [Andrew Or] Add tests for scheduling with executor limit
a82e907 [Andrew Or] Fix comments
0a8be79 [Andrew Or] Simplify logic by removing the worker blacklist
b7742af [Andrew Or] Merge branch 'master' of github.com:apache/spark into standalone-da
2eb5f3f [Andrew Or] Merge branch 'master' of github.com:apache/spark into standalone-da
1334e9a [Andrew Or] Fix MiMa
32abe44 [Andrew Or] Fix style
58cb06f [Andrew Or] Privatize worker blacklist for cleanliness
42ac215 [Andrew Or] Clean up comments and rewrite code for readability
49702d1 [Andrew Or] Clean up shuffle files after application exits
80047aa [Andrew Or] First working implementation
Diffstat (limited to 'yarn')
0 files changed, 0 insertions, 0 deletions