aboutsummaryrefslogtreecommitdiff
path: root/ec2/deploy.generic/root
diff options
context:
space:
mode:
authorNicholas Chammas <nicholas.chammas@gmail.com>2015-03-10 10:58:31 +0000
committerSean Owen <sowen@cloudera.com>2015-03-10 10:58:31 +0000
commitd14df06c05a6228fd6522914c39aa75898eddfc1 (patch)
tree5e5cf41da06ee017762924e4bd4a991b113da4c3 /ec2/deploy.generic/root
parentc4c4b07bf61cab01d92fde4f902d8c06abdce240 (diff)
downloadspark-d14df06c05a6228fd6522914c39aa75898eddfc1.tar.gz
spark-d14df06c05a6228fd6522914c39aa75898eddfc1.tar.bz2
spark-d14df06c05a6228fd6522914c39aa75898eddfc1.zip
[SPARK-6191] [EC2] Generalize ability to download libs
Right now we have a method to specifically download boto. This PR generalizes it so it's easy to download additional libraries if we want. For example, adding new external libraries for spark-ec2 is now as simple as: ```python external_libs = [ { "name": "boto", "version": "2.34.0", "md5": "5556223d2d0cc4d06dd4829e671dcecd" }, { "name": "PyYAML", "version": "3.11", "md5": "f50e08ef0fe55178479d3a618efe21db" }, { "name": "argparse", "version": "1.3.0", "md5": "9bcf7f612190885c8c85e30ba41db3c7" } ] ``` Likely use cases: * Downloading PyYAML to allow spark-ec2 configs to be persisted as a YAML file. ([SPARK-925](https://issues.apache.org/jira/browse/SPARK-925)) * Downloading argparse to clean up / modernize our option parsing. First run output, with PyYAML and argparse added just for demonstration purposes: ```shell $ ./spark-ec2 --version Downloading external libraries that spark-ec2 needs from PyPI to /path/to/spark/ec2/lib... This should be a one-time operation. - Downloading boto... - Finished downloading boto. - Downloading PyYAML... - Finished downloading PyYAML. - Downloading argparse... - Finished downloading argparse. spark-ec2 1.2.1 ``` Output thereafter: ```shell $ ./spark-ec2 --version spark-ec2 1.2.1 ``` Author: Nicholas Chammas <nicholas.chammas@gmail.com> Closes #4919 from nchammas/setup-ec2-libs and squashes the following commits: a077955 [Nicholas Chammas] print default region c95fb7d [Nicholas Chammas] to docstring 5448845 [Nicholas Chammas] remove libs added for demo purposes 60d8c23 [Nicholas Chammas] generalize ability to download libs
Diffstat (limited to 'ec2/deploy.generic/root')
0 files changed, 0 insertions, 0 deletions