| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Show the region for the different messages displayed by get_existing_cluster(): The search, found and error messages.
Author: Miguel Peralvo <miguel.peralvo@gmail.com>
Closes #4457 from MiguelPeralvo/patch-2 and squashes the following commits:
a5514c8 [Miguel Peralvo] Update spark_ec2.py
0a837b0 [Miguel Peralvo] Update spark_ec2.py
3923f36 [Miguel Peralvo] Update spark_ec2.py
4ecd9f9 [Miguel Peralvo] [SPARK-5668] Display region in spark_ec2.py get_existing_cluster()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Addresses [SPARK-1805](https://issues.apache.org/jira/browse/SPARK-1805), though doesn't resolve it completely.
Error out quickly if the user asks for the master and slaves to have different AMI virtualization types, since we don't currently support that.
In addition to that, we print warnings if the inputted instance types are not recognized, though I would prefer if we errored out. Elsewhere in the script it seems [we allow unrecognized instance types](https://github.com/apache/spark/blob/5de14cc2763a8211f77eeb55940dec025822eb78/ec2/spark_ec2.py#L331), though I think we should remove that.
It's messy, but it should serve us until we enhance spark-ec2 to support clusters with mixed virtualization types.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #4455 from nchammas/ec2-master-slave-different-virtualization and squashes the following commits:
ce28609 [Nicholas Chammas] fix style
b0adba0 [Nicholas Chammas] validate input instance types
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
spark_ec2.py
and by extension, the ami-list
Useful for using alternate spark-ec2 repos or branches.
Author: Florian Verhein <florian.verhein@gmail.com>
Closes #4385 from florianverhein/master and squashes the following commits:
7e2b4be [Florian Verhein] [SPARK-5611] [EC2] typo
8b653dc [Florian Verhein] [SPARK-5611] [EC2] Enforce only supporting spark-ec2 forks from github, log improvement
bc4b0ed [Florian Verhein] [SPARK-5611] allow spark-ec2 repos with different names
8b5c551 [Florian Verhein] improve option naming, fix logging, fix lint failing, add guard to enforce spark-ec2
7724308 [Florian Verhein] [SPARK-5611] [EC2] fixes
b42b68c [Florian Verhein] [SPARK-5611] [EC2] Allow spark-ec2 repo and branch to be set on CLI of spark_ec2.py
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If there is some fatal problem with launching a cluster, `spark-ec2` just hangs without giving the user useful feedback on what the problem is.
This PR exposes the output of the SSH calls to the user if the SSH test fails during cluster launch for any reason but the instance status checks are all green. It also removes the growing trail of dots while waiting in favor of a fixed 3 dots.
For example:
```
$ ./ec2/spark-ec2 -k key -i /incorrect/path/identity.pem --instance-type m3.medium --slaves 1 --zone us-east-1c launch "spark-test"
Setting up security groups...
Searching for existing cluster spark-test...
Spark AMI: ami-35b1885c
Launching instances...
Launched 1 slaves in us-east-1c, regid = r-7dadd096
Launched master in us-east-1c, regid = r-fcadd017
Waiting for cluster to enter 'ssh-ready' state...
Warning: SSH connection error. (This could be temporary.)
Host: 127.0.0.1
SSH return code: 255
SSH output: Warning: Identity file /incorrect/path/identity.pem not accessible: No such file or directory.
Warning: Permanently added '127.0.0.1' (RSA) to the list of known hosts.
Permission denied (publickey).
```
This should give users enough information when some unrecoverable error occurs during launch so they can know to abort the launch. This will help avoid situations like the ones reported [here on Stack Overflow](http://stackoverflow.com/q/28002443/) and [here on the user list](http://mail-archives.apache.org/mod_mbox/spark-user/201501.mbox/%3C1422323829398-21381.postn3.nabble.com%3E), where the users couldn't tell what the problem was because it was being hidden by `spark-ec2`.
This is a usability improvement that should be backported to 1.2.
Resolves [SPARK-5473](https://issues.apache.org/jira/browse/SPARK-5473).
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #4262 from nchammas/expose-ssh-failure and squashes the following commits:
8bda6ed [Nicholas Chammas] default to print SSH output
2b92534 [Nicholas Chammas] show SSH output after status check pass
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Check the mode of private key file.
Author: liuchang0812 <liuchang0812@gmail.com>
Closes #4162 from Liuchang0812/ec2-script and squashes the following commits:
fc37355 [liuchang0812] quota file name
01ed464 [liuchang0812] more output
ce2a207 [liuchang0812] pep8
f44efd2 [liuchang0812] move code to real_main
8475a54 [liuchang0812] fix bug
cd61a1a [liuchang0812] import stat
c106cb2 [liuchang0812] fix trivis bug
89c9953 [liuchang0812] more output about checking private key
1177a90 [liuchang0812] remove commet
41188ab [liuchang0812] check the mode of private key
|
|
|
|
|
|
|
|
|
|
| |
See https://issues.apache.org/jira/browse/SPARK-5403
Author: Grzegorz Dubicki <grzegorz.dubicki@gmail.com>
Closes #4196 from grzegorz-dubicki/SPARK-5403 and squashes the following commits:
a7d863f [Grzegorz Dubicki] Resolve start command hanging issue
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The boto API doesn't support tag EC2 instances in the same call that launches them.
We add a five-second wait so EC2 has enough time to propagate the information so that
the tagging can succeed.
Author: GenTang <gen.tang86@gmail.com>
Author: Gen TANG <gen.tang86@gmail.com>
Closes #3986 from GenTang/spark-4983 and squashes the following commits:
13e257d [Gen TANG] modification of comments
47f06755 [GenTang] print the information
ab7a931 [GenTang] solve the issus spark-4983 by inserting waiting time
3179737 [GenTang] Revert "handling exceptions about adding tags to ec2"
6a8b53b [GenTang] Revert "the improvement of exception handling"
13e97a6 [GenTang] Revert "typo"
63fd360 [GenTang] typo
692fc2b [GenTang] the improvement of exception handling
6adcf6d [GenTang] handling exceptions about adding tags to ec2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every proper command line tool should include a `--version` option or something similar.
This PR adds this to `spark-ec2` using the standard functionality provided by `optparse`.
One thing we don't do here is follow the Python convention of setting `__version__`, since it seems awkward given how `spark-ec2` is laid out.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #4414 from nchammas/spark-ec2-show-version and squashes the following commits:
914cab5 [Nicholas Chammas] add version info
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes [SPARK-5434](https://issues.apache.org/jira/browse/SPARK-5434).
Simple demonstration of the problem and the fix:
```
$ spacey_path="/path/with some/spaces"
$ dirname $spacey_path
usage: dirname path
$ echo $?
1
$ dirname "$spacey_path"
/path/with some
$ echo $?
0
```
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #4224 from nchammas/patch-1 and squashes the following commits:
960711a [Nicholas Chammas] [EC2] Preserve spaces in EC2 path
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I moved the Spark-Shark version map [to the wiki](https://cwiki.apache.org/confluence/display/SPARK/Spark-Shark+version+mapping).
This PR has a [matching PR in mesos/spark-ec2](https://github.com/mesos/spark-ec2/pull/89).
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #3939 from nchammas/remove-shark and squashes the following commits:
66e0841 [Nicholas Chammas] fix style
ceeab85 [Nicholas Chammas] show default Spark GitHub repo
7270126 [Nicholas Chammas] validate Spark hashes
db4935d [Nicholas Chammas] validate spark version upfront
fc0d5b9 [Nicholas Chammas] remove Shark
|
|
|
|
|
|
|
|
|
|
| |
Going forward, we'll use matching branch names across the mesos/spark-ec2 and apache/spark repositories, per [the discussion here](https://github.com/mesos/spark-ec2/pull/85#issuecomment-68069589).
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #3804 from nchammas/patch-2 and squashes the following commits:
cd2c0d4 [Nicholas Chammas] [EC2] Update mesos/spark-ec2 branch to branch-1.3
|
|
|
|
|
|
|
|
|
|
|
| |
Now that 1.2.0 is out, let's update the default Spark version.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #3793 from nchammas/patch-1 and squashes the following commits:
3255832 [Nicholas Chammas] add 1.2.0 version to Spark-Shark map
ec0e904 [Nicholas Chammas] [EC2] Update default Spark version to 1.2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
instead of packaging it
This patch upgrades `spark-ec2`'s Boto version to 2.34.0, since this is blocking several features. Newer versions of Boto don't work properly when they're loaded from a zipfile since they try to read a JSON file from a path relative to the Boto library sources.
Therefore, this patch also changes spark-ec2 to automatically download Boto from PyPi if it's not present in `SPARK_EC2_DIR/lib`, similar to what we do in the `sbt/sbt` script. This shouldn't ben an issue for users since they already need to have an internet connection to launch an EC2 cluster. By performing the downloading in spark_ec2.py instead of the Bash script, this should also work for Windows users.
I've tested this with Python 2.6, too.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #3737 from JoshRosen/update-boto and squashes the following commits:
0aa43cc [Josh Rosen] Remove unused setup_standalone_cluster() method.
f02935d [Josh Rosen] Enable Python deprecation warnings and fix one Boto warning:
587ae89 [Josh Rosen] [SPARK-4890] Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
spark_ec2
Placement groups are cool and all the cool kids are using them. Lets add support for them to spark_ec2.py because I'm lazy
Author: Holden Karau <holden@pigscanfly.ca>
Closes #3623 from holdenk/SPARK-4767-add-support-for-launching-in-a-specified-placement-group-to-spark-ec2-scripts and squashes the following commits:
111a5fd [Holden Karau] merge in master
70ace25 [Holden Karau] Placement groups are cool and all the cool kids are using them. Lets add support for them to spark_ec2.py because I'm lazy
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Based on this gist:
https://gist.github.com/amar-analytx/0b62543621e1f246c0a2
We use security group ids instead of security group to get around this issue:
https://github.com/boto/boto/issues/350
Author: Mike Jennings <mvj101@gmail.com>
Author: Mike Jennings <mvj@google.com>
Closes #2872 from mvj101/SPARK-3405 and squashes the following commits:
be9cb43 [Mike Jennings] `pep8 spark_ec2.py` runs cleanly.
4dc6756 [Mike Jennings] Remove duplicate comment
731d94c [Mike Jennings] Update for code review.
ad90a36 [Mike Jennings] Merge branch 'master' of https://github.com/apache/spark into SPARK-3405
1ebffa1 [Mike Jennings] Merge branch 'master' into SPARK-3405
52aaeec [Mike Jennings] [SPARK-3405] add subnet-id and vpc-id options to spark_ec2.py
|
|
|
|
|
|
|
|
|
|
| |
The current get_existing_cluster() function would only find an instance belonged to a cluster if the instance's security groups == cluster_name + "-master" (or "-slaves"). This fix allows for multiple security groups by checking if the cluster_name + "-master" security group is in the list of groups for a particular instance.
Author: alexdebrie <alexdebrie1@gmail.com>
Closes #3596 from alexdebrie/master and squashes the following commits:
9d51232 [alexdebrie] Fix get_existing_cluster() function with multiple security groups
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This PR re-introduces [0e648bc](https://github.com/apache/spark/commit/0e648bc2bedcbeb55fce5efac04f6dbad9f063b4) from PR #2339, which somehow never made it into the codebase.
Additionally, it removes a now-unnecessary linear backoff on the SSH checks since we are blocking on EC2 status checks before testing SSH.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #3195 from nchammas/remove-ec2-ssh-backoff and squashes the following commits:
efb29e1 [Nicholas Chammas] Revert "Remove linear backoff."
ef3ca99 [Nicholas Chammas] reuse conn
adb4eaa [Nicholas Chammas] Remove linear backoff.
55caa24 [Nicholas Chammas] Check EC2 status checks before SSH.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This looks like a one-liner, so I took a shot at it. There can be no fixed default availability zone since the names are different per region. But the default behavior can be documented:
```
if opts.zone == "":
opts.zone = random.choice(conn.get_all_zones()).name
```
Author: Sean Owen <sowen@cloudera.com>
Closes #3454 from srowen/SPARK-1450 and squashes the following commits:
9193cf3 [Sean Owen] Document that --zone defaults to a single random zone
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This PR reverts changes related to tag-based cluster membership. As discussed in SPARK-3332, we didn't figure out a safe strategy to use tags to determine cluster membership, because tagging is not atomic. The following changes are reverted:
SPARK-2333: 94053a7b766788bb62e2dbbf352ccbcc75f71fc0
SPARK-3213: 7faf755ae4f0cf510048e432340260a6e609066d
SPARK-3608: 78d4220fa0bf2f9ee663e34bbf3544a5313b02f0.
I tested launch, login, and destroy. It is easy to check the diff by comparing it to Josh's patch for branch-1.1:
https://github.com/apache/spark/pull/2225/files
JoshRosen I sent the PR to master. It might be easier for us to keep master and branch-1.2 the same at this time. We can always re-apply the patch once we figure out a stable solution.
Author: Xiangrui Meng <meng@databricks.com>
Closes #3453 from mengxr/SPARK-4509 and squashes the following commits:
f0b708b [Xiangrui Meng] revert 94053a7b766788bb62e2dbbf352ccbcc75f71fc0
4298ea5 [Xiangrui Meng] revert 7faf755ae4f0cf510048e432340260a6e609066d
35963a1 [Xiangrui Meng] Revert "SPARK-3608 Break if the instance tag naming succeeds"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This issue was uncovered after [this discussion](https://issues.apache.org/jira/browse/SPARK-3398?focusedCommentId=14187471&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14187471).
Don't change the working directory on the user. This breaks relative paths the user may pass in, e.g., for the SSH identity file.
```
./ec2/spark-ec2 -i ../my.pem
```
This patch will preserve the user's current working directory and allow calls like the one above to work.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #2988 from nchammas/spark-ec2-cwd and squashes the following commits:
f3850b5 [Nicholas Chammas] pep8 fix
fbc20c7 [Nicholas Chammas] revert to old commenting style
752f958 [Nicholas Chammas] specify deploy.generic path absolutely
bcdf6a5 [Nicholas Chammas] fix typo
77871a2 [Nicholas Chammas] add clarifying comment
ce071fc [Nicholas Chammas] don't change working dir
|
|
|
|
|
|
|
|
|
|
| |
We reference a specific branch in two places. This patch makes it one place.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #3008 from nchammas/mesos-spark-ec2-branch and squashes the following commits:
10a6089 [Nicholas Chammas] factor out mess spark-ec2 branch
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of waiting arbitrary amounts of time for the cluster to reach a specific state, this patch lets `spark-ec2` explicitly wait for a cluster to reach a desired state.
This is useful in a couple of situations:
* The cluster is launching and you want to wait until SSH is available before installing stuff.
* The cluster is being terminated and you want to wait until all the instances are terminated before trying to delete security groups.
This patch removes the need for the `--wait` option and removes some of the time-based retry logic that was being used.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #2339 from nchammas/spark-ec2-wait-properly and squashes the following commits:
43a69f0 [Nicholas Chammas] short-circuit SSH check; linear backoff
9a9e035 [Nicholas Chammas] remove extraneous comment
26c5ed0 [Nicholas Chammas] replace print with write()
bb67c06 [Nicholas Chammas] deprecate wait option; remove dead code
7969265 [Nicholas Chammas] fix long line (PEP 8)
126e4cf [Nicholas Chammas] wait for specific cluster states
|
|
|
|
|
|
|
|
|
|
| |
Similar to the work done in #2571, this PR just sorts the remaining manually-inputted dicts in the EC2 script so they are easier to maintain.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #2578 from nchammas/ec2-dict-sort and squashes the following commits:
f55c692 [Nicholas Chammas] sort long dictionaries
|
|
|
|
|
|
|
|
|
|
|
|
| |
Minor fixes:
* Remove unnecessary parens (Python style)
* Sort `disks_by_instance` dict and remove duplicate `t1.micro` key
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #2571 from nchammas/ec2-polish and squashes the following commits:
9d203d5 [Nicholas Chammas] paren and dict cleanup
|
|
|
|
|
|
|
|
|
|
| |
This brings the master branch in sync with branch-1.1
Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
Closes #2510 from shivaram/spark-ec2-version and squashes the following commits:
bb0dd16 [Shivaram Venkataraman] Set EC2 version to 1.1.0 and update version map
|
|
|
|
|
|
|
|
| |
Author: Vida Ha <vida@databricks.com>
Closes #2466 from vidaha/vida/spark-3608 and squashes the following commits:
9509776 [Vida Ha] Break if the instance tag naming succeeds
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When deploying to AWS, there is additional configuration that is required to read S3 files. EMR creates it automatically, there is no reason that the Spark EC2 script shouldn't.
This PR requires a corresponding PR to the mesos/spark-ec2 to be merged, as it gets cloned in the process of setting up machines: https://github.com/mesos/spark-ec2/pull/58
Author: Dan Osipov <daniil.osipov@shazam.com>
Closes #1120 from danosipov/s3_credentials and squashes the following commits:
758da8b [Dan Osipov] Modify documentation to include the new parameter
71fab14 [Dan Osipov] Use a parameter --copy-aws-credentials to enable S3 credential deployment
7e0da26 [Dan Osipov] Get AWS credentials out of boto connection instance
39bdf30 [Dan Osipov] Add S3 configuration parameters to the EC2 deploy scripts
|
|
|
|
|
|
|
|
|
|
| |
Tested on a real cluster.
Author: Reynold Xin <rxin@apache.org>
Closes #2404 from rxin/ec2-reboot-slaves and squashes the following commits:
00a2dbd [Reynold Xin] Allow rebooting slaves.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This PR makes two minor changes to the `spark-ec2` script:
1. The script's input parameter default values are duplicated into the help text. This is unnecessary. This PR replaces the duplicated info with the appropriate `optparse` placeholder.
2. The default Spark version currently needs to be updated by hand during each release, which is known to be a faulty process. This PR places that default value in an easy-to-spot place.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #2290 from nchammas/spark-ec2-default-version and squashes the following commits:
0c6d3bb [Nicholas Chammas] don't duplicate default values
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This PR resolves [SPARK-3361](https://issues.apache.org/jira/browse/SPARK-3361) by expanding the PEP 8 checks to cover the remaining Python code base:
* The EC2 script
* All Python / PySpark examples
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #2297 from nchammas/pep8-rulez and squashes the following commits:
1e5ac9a [Nicholas Chammas] PEP 8 fixes to Python examples
c3dbeff [Nicholas Chammas] PEP 8 fixes to EC2 script
65ef6e8 [Nicholas Chammas] expand PEP 8 checks
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Please merge this at the same time as https://github.com/mesos/spark-ec2/pull/66
Author: Reynold Xin <rxin@apache.org>
Closes #2260 from rxin/ec2-ebs-vol and squashes the following commits:
b9527d9 [Reynold Xin] Removed io1 ebs type.
bf9c403 [Reynold Xin] Made EBS volume type configurable.
c8e25ea [Reynold Xin] Support up to 8 EBS volumes.
adf4f2e [Reynold Xin] Revert git repo change.
020c542 [Reynold Xin] [SPARK-3391] Support attaching more than 1 EBS volumes.
|
|
|
|
|
|
|
|
|
|
|
|
| |
During regression tests of Spark 1.1 we discovered perf issues with
PVM instances when running PySpark. This reverts a change added in #1156
which changed the default type for m3 instances to PVM.
Author: Patrick Wendell <pwendell@gmail.com>
Closes #2244 from pwendell/ec2-hvm and squashes the following commits:
1342d7e [Patrick Wendell] SPARK-3358: [EC2] Switch back to HVM instances for m3.X.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
On `m3.2xlarge` instances the 2x80GB SSDs are inaccessible if not added to the block device mapping when the instance is created. They work when added with this patch. I have not tested this with other instance types, and I do not know much about this script and EC2 deployment in general. Maybe this code needs to depend on the instance type.
The requirement for this mapping is described in the AWS docs at:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#InstanceStore_UsageScenarios
"For M3 instances, you must specify instance store volumes in the block
device mapping for the instance. When you launch an M3 instance, we
ignore any instance store volumes specified in the block device mapping
for the AMI."
Author: Daniel Darabos <darabos.daniel@gmail.com>
Closes #2081 from darabos/patch-1 and squashes the following commits:
1ceb2c8 [Daniel Darabos] Use %d string interpolation instead of {}.
a1854d7 [Daniel Darabos] Only specify ephemeral device mapping for M3.
e0d9e37 [Daniel Darabos] Create ephemeral device mapping based on get_num_disks().
6b116a6 [Daniel Darabos] Add SSDs to block device mapping
|
|
|
|
|
|
|
|
|
|
|
|
| |
"Launch More like this"
... copy the spark_cluster_tag from a spot instance requests over to the instances.
Author: Vida Ha <vida@databricks.com>
Closes #2163 from vidaha/vida/spark-3213 and squashes the following commits:
5070a70 [Vida Ha] Spark-3214 Fix issue with spark-ec2 not detecting slaves created with 'Launch More Like This' and using Spot Requests
|
|
|
|
|
|
|
|
| |
Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com>
Closes #2162 from douglaz/user_data_master and squashes the following commits:
10d15f6 [Allan Douglas R. de Oliveira] Give user data also to the master
|
|
|
|
|
|
|
|
|
|
| |
Adds the --authorized-address and --additional-security-group options as explained in the issue.
Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com>
Closes #2088 from douglaz/configurable_sg and squashes the following commits:
e3e48ca [Allan Douglas R. de Oliveira] Adds the option to specify the address authorized to access the SG and another option to provide an additional existing SG
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Uses the name tag to identify machines in a cluster.
- Allows overriding the security group name so it doesn't need to coincide with the cluster name.
- Outputs the request id's of up to 10 pending spot instance requests.
Author: Vida Ha <vida@databricks.com>
Closes #1899 from vidaha/vida/ec2-reuse-security-group and squashes the following commits:
c80d5c3 [Vida Ha] wrap retries in a try catch block
b2989d5 [Vida Ha] SPARK-2333: spark_ec2 script should allow option for existing security group
|
|
|
|
|
|
|
|
| |
Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com>
Closes #1186 from douglaz/spark_ec2_user_data and squashes the following commits:
94a36f9 [Allan Douglas R. de Oliveira] Added user data option to EC2 script
|
|
|
|
|
|
|
|
|
|
|
|
| |
New t2 instance types require HVM amis, bailout assumption of pvm
causes failures when using t2 instance types.
Author: Basit Mustafa <basitmustafa@computes-things-for-basit.local>
Closes #1446 from 24601/master and squashes the following commits:
01fe128 [Basit Mustafa] Makin' it pretty
392a95e [Basit Mustafa] Added t2 instance types
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Security groups created by `spark-ec2` do not prepend “spark-“ to the
name.
Since naming the instances themselves is new to `spark-ec2`, it’s better
to change that pattern to match the existing naming pattern for the
security groups, rather than the other way around.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Author: nchammas <nicholas.chammas@gmail.com>
Closes #1344 from nchammas/master and squashes the following commits:
f7e4581 [Nicholas Chammas] unrelated pep8 fix
a36eed0 [Nicholas Chammas] name ec2 instances and security groups consistently
de7292a [nchammas] Merge pull request #4 from apache/master
2e4fe00 [nchammas] Merge pull request #3 from apache/master
89fde08 [nchammas] Merge pull request #2 from apache/master
69f6e22 [Nicholas Chammas] PEP8 fixes
2627247 [Nicholas Chammas] broke up lines before they hit 100 chars
6544b7e [Nicholas Chammas] [SPARK-2065] give launched instances names
69da6cf [nchammas] Merge pull request #1 from apache/master
|
|
|
|
|
|
|
|
|
|
| |
Right now I have to open it manually
Author: Andrew Or <andrewor14@gmail.com>
Closes #1296 from andrewor14/hist-serv-port and squashes the following commits:
8895a1f [Andrew Or] Add default history server port to ec2 script
|
|
|
|
|
|
|
|
|
|
| |
Fixed a problem in previous file in which some information regarding AWS instance types were wrong. Such information was updated base upon current AWS EC2 data.
Author: Zichuan Ye <jerry@tangentds.com>
Closes #1156 from jerry86/master and squashes the following commits:
ff36e95 [Zichuan Ye] Fixing AWS instance type information based upon current EC2 data
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Will list the EC2 instances before detroying the cluster.
This was added because it can be scary to destroy EC2
instances without knowing which one will be impacted.
Author: Jean-Martin Archer <jeanmartin.archer@pulseenergy.com>
This patch had conflicts when merged, resolved by
Committer: Patrick Wendell <pwendell@gmail.com>
Closes #270 from j-martin/master and squashes the following commits:
826455f [Jean-Martin Archer] [SPARK-2611] Implementing recommendations
27b0a36 [Jean-Martin Archer] Listing of instances to be terminated before the prompt Will list the EC2 instances before detroying the cluster. This was added because it can be scary to destroy EC2 instances without knowing which one will be impacted.
|
|
|
|
|
|
|
|
|
|
| |
To preserve quoted command line args (in case options have space in them).
Author: Ori Kremer <ori.kremer@gmail.com>
Closes #1169 from orikremer/quote_cmd_line_args and squashes the following commits:
67e2aa1 [Ori Kremer] quote command line args
|
|
|
|
|
|
|
|
|
|
| |
This patch should have qualified the use of PIPE. This needs to be back ported into 0.9 and 1.0.
Author: Patrick Wendell <pwendell@gmail.com>
Closes #1108 from pwendell/hotfix and squashes the following commits:
711c58d [Patrick Wendell] HOTFIX: bug caused by #941
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
https://issues.apache.org/jira/browse/SPARK-1990
There were some posts on the lists that spark-ec2 does not work with Python 2.6. In addition, we should check the Python version at the top of the script and exit if it's too old
Author: Anant <anant.asty@gmail.com>
Closes #941 from anantasty/SPARK-1990 and squashes the following commits:
4ca441d [Anant] Implmented check_optput withinthe module to work with python 2.6
c6ed85c [Anant] added compatibility for python 2.6 for ssh_read command
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This update resolves [SPARK-2065](https://issues.apache.org/jira/browse/SPARK-2065). It gives launched EC2 instances descriptive names by using instance tags. Launched instances now show up in the EC2 console with these names.
I used `format()` with named parameters, which I believe is the recommended practice for string formatting in Python, but which doesn’t seem to be used elsewhere in the script.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Author: nchammas <nicholas.chammas@gmail.com>
Closes #1043 from nchammas/master and squashes the following commits:
69f6e22 [Nicholas Chammas] PEP8 fixes
2627247 [Nicholas Chammas] broke up lines before they hit 100 chars
6544b7e [Nicholas Chammas] [SPARK-2065] give launched instances names
69da6cf [nchammas] Merge pull request #1 from apache/master
|
|
|
|
|
|
|
|
| |
Author: Varakhedi Sujeet <svarakhedi@gopivotal.com>
Closes #960 from sujeetv/ec2-r3 and squashes the following commits:
3cb9fd5 [Varakhedi Sujeet] SPARK-1790: Update EC2 scripts to support r3 instance
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The change was previously committed only to branch-1.0 as part of https://github.com/apache/spark/commit/a34e6fda1d6fb8e769c21db70845f1a6dde968d8
Author: Aaron Davidson <aaron@databricks.com>
This patch had conflicts when merged, resolved by
Committer: Patrick Wendell <pwendell@gmail.com>
Closes #938 from aarondav/sparkec2 and squashes the following commits:
067cc31 [Aaron Davidson] Update spark-ec2 scripts for 1.0.0 on master
|