diff options
| author | Nicholas Chammas <nicholas.chammas@gmail.com> | 2014-10-07 16:54:32 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@apache.org> | 2014-10-07 16:54:32 -0700 |
| commit | 5912ca67140eed5dea66745aa3af4febdbd80781 (patch) | |
| tree | c01f62760a782ac3d26f4a1eda118d344cfe5a95 /python/pyspark/context.py | |
| parent | b32bb72e812731d28bf05f2145314c63806f3335 (diff) | |
| download | spark-5912ca67140eed5dea66745aa3af4febdbd80781.tar.gz spark-5912ca67140eed5dea66745aa3af4febdbd80781.tar.bz2 spark-5912ca67140eed5dea66745aa3af4febdbd80781.zip | |
[SPARK-3398] [EC2] Have spark-ec2 intelligently wait for specific cluster states
Instead of waiting arbitrary amounts of time for the cluster to reach a specific state, this patch lets `spark-ec2` explicitly wait for a cluster to reach a desired state.
This is useful in a couple of situations:
* The cluster is launching and you want to wait until SSH is available before installing stuff.
* The cluster is being terminated and you want to wait until all the instances are terminated before trying to delete security groups.
This patch removes the need for the `--wait` option and removes some of the time-based retry logic that was being used.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #2339 from nchammas/spark-ec2-wait-properly and squashes the following commits:
43a69f0 [Nicholas Chammas] short-circuit SSH check; linear backoff
9a9e035 [Nicholas Chammas] remove extraneous comment
26c5ed0 [Nicholas Chammas] replace print with write()
bb67c06 [Nicholas Chammas] deprecate wait option; remove dead code
7969265 [Nicholas Chammas] fix long line (PEP 8)
126e4cf [Nicholas Chammas] wait for specific cluster states
Diffstat (limited to 'python/pyspark/context.py')
0 files changed, 0 insertions, 0 deletions