diff options
| author | Mike Jennings <mvj101@gmail.com> | 2014-12-16 12:13:21 -0800 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2014-12-16 12:13:21 -0800 |
| commit | d12c0711faa3d4333513fcbbbee4868bcb784a26 (patch) | |
| tree | e4fd21418ffa640b9bd0a668d5c098f58c15c512 /docs/ec2-scripts.md | |
| parent | cb484474934d664000df3d63a326bcd6b12f2f09 (diff) | |
| download | spark-d12c0711faa3d4333513fcbbbee4868bcb784a26.tar.gz spark-d12c0711faa3d4333513fcbbbee4868bcb784a26.tar.bz2 spark-d12c0711faa3d4333513fcbbbee4868bcb784a26.zip | |
[SPARK-3405] add subnet-id and vpc-id options to spark_ec2.py
Based on this gist:
https://gist.github.com/amar-analytx/0b62543621e1f246c0a2
We use security group ids instead of security group to get around this issue:
https://github.com/boto/boto/issues/350
Author: Mike Jennings <mvj101@gmail.com>
Author: Mike Jennings <mvj@google.com>
Closes #2872 from mvj101/SPARK-3405 and squashes the following commits:
be9cb43 [Mike Jennings] `pep8 spark_ec2.py` runs cleanly.
4dc6756 [Mike Jennings] Remove duplicate comment
731d94c [Mike Jennings] Update for code review.
ad90a36 [Mike Jennings] Merge branch 'master' of https://github.com/apache/spark into SPARK-3405
1ebffa1 [Mike Jennings] Merge branch 'master' into SPARK-3405
52aaeec [Mike Jennings] [SPARK-3405] add subnet-id and vpc-id options to spark_ec2.py
Diffstat (limited to 'docs/ec2-scripts.md')
| -rw-r--r-- | docs/ec2-scripts.md | 19 |
1 files changed, 19 insertions, 0 deletions
diff --git a/docs/ec2-scripts.md b/docs/ec2-scripts.md index ed51d0abb3..d50f445d7e 100644 --- a/docs/ec2-scripts.md +++ b/docs/ec2-scripts.md @@ -94,6 +94,25 @@ another. permissions on your private key file, you can run `launch` with the `--resume` option to restart the setup process on an existing cluster. +# Launching a Cluster in a VPC + +- Run + `./spark-ec2 -k <keypair> -i <key-file> -s <num-slaves> --vpc-id=<vpc-id> --subnet-id=<subnet-id> launch <cluster-name>`, + where `<keypair>` is the name of your EC2 key pair (that you gave it + when you created it), `<key-file>` is the private key file for your + key pair, `<num-slaves>` is the number of slave nodes to launch (try + 1 at first), `<vpc-id>` is the name of your VPC, `<subnet-id>` is the + name of your subnet, and `<cluster-name>` is the name to give to your + cluster. + + For example: + + ```bash + export AWS_SECRET_ACCESS_KEY=AaBbCcDdEeFGgHhIiJjKkLlMmNnOoPpQqRrSsTtU +export AWS_ACCESS_KEY_ID=ABCDEFG1234567890123 +./spark-ec2 --key-pair=awskey --identity-file=awskey.pem --region=us-west-1 --zone=us-west-1a --vpc-id=vpc-a28d24c7 --subnet-id=subnet-4eb27b39 --spark-version=1.1.0 launch my-spark-cluster + ``` + # Running Applications - Go into the `ec2` directory in the release of Spark you downloaded. |