diff options
author | Nicholas Chammas <nicholas.chammas@gmail.com> | 2015-03-10 10:58:31 +0000 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2015-03-10 10:58:31 +0000 |
commit | d14df06c05a6228fd6522914c39aa75898eddfc1 (patch) | |
tree | 5e5cf41da06ee017762924e4bd4a991b113da4c3 /streaming | |
parent | c4c4b07bf61cab01d92fde4f902d8c06abdce240 (diff) | |
download | spark-d14df06c05a6228fd6522914c39aa75898eddfc1.tar.gz spark-d14df06c05a6228fd6522914c39aa75898eddfc1.tar.bz2 spark-d14df06c05a6228fd6522914c39aa75898eddfc1.zip |
[SPARK-6191] [EC2] Generalize ability to download libs
Right now we have a method to specifically download boto. This PR generalizes it so it's easy to download additional libraries if we want.
For example, adding new external libraries for spark-ec2 is now as simple as:
```python
external_libs = [
{
"name": "boto",
"version": "2.34.0",
"md5": "5556223d2d0cc4d06dd4829e671dcecd"
},
{
"name": "PyYAML",
"version": "3.11",
"md5": "f50e08ef0fe55178479d3a618efe21db"
},
{
"name": "argparse",
"version": "1.3.0",
"md5": "9bcf7f612190885c8c85e30ba41db3c7"
}
]
```
Likely use cases:
* Downloading PyYAML to allow spark-ec2 configs to be persisted as a YAML file. ([SPARK-925](https://issues.apache.org/jira/browse/SPARK-925))
* Downloading argparse to clean up / modernize our option parsing.
First run output, with PyYAML and argparse added just for demonstration purposes:
```shell
$ ./spark-ec2 --version
Downloading external libraries that spark-ec2 needs from PyPI to /path/to/spark/ec2/lib...
This should be a one-time operation.
- Downloading boto...
- Finished downloading boto.
- Downloading PyYAML...
- Finished downloading PyYAML.
- Downloading argparse...
- Finished downloading argparse.
spark-ec2 1.2.1
```
Output thereafter:
```shell
$ ./spark-ec2 --version
spark-ec2 1.2.1
```
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes #4919 from nchammas/setup-ec2-libs and squashes the following commits:
a077955 [Nicholas Chammas] print default region
c95fb7d [Nicholas Chammas] to docstring
5448845 [Nicholas Chammas] remove libs added for demo purposes
60d8c23 [Nicholas Chammas] generalize ability to download libs
Diffstat (limited to 'streaming')
0 files changed, 0 insertions, 0 deletions