diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2014-12-19 17:02:37 -0800 |
|---|---|---|
| committer | Josh Rosen <joshrosen@databricks.com> | 2014-12-19 17:02:37 -0800 |
| commit | c28083f468685d468c4daa5ce336ef04fbec3144 (patch) | |
| tree | 56972f9c317ee1374478eafae08e834dfa7e48fa /ec2/spark-ec2 | |
| parent | 7981f969762e77f1752ef8f86c546d4fc32a1a4f (diff) | |
| download | spark-c28083f468685d468c4daa5ce336ef04fbec3144.tar.gz spark-c28083f468685d468c4daa5ce336ef04fbec3144.tar.bz2 spark-c28083f468685d468c4daa5ce336ef04fbec3144.zip | |
[SPARK-4890] Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it
This patch upgrades `spark-ec2`'s Boto version to 2.34.0, since this is blocking several features. Newer versions of Boto don't work properly when they're loaded from a zipfile since they try to read a JSON file from a path relative to the Boto library sources.
Therefore, this patch also changes spark-ec2 to automatically download Boto from PyPi if it's not present in `SPARK_EC2_DIR/lib`, similar to what we do in the `sbt/sbt` script. This shouldn't ben an issue for users since they already need to have an internet connection to launch an EC2 cluster. By performing the downloading in spark_ec2.py instead of the Bash script, this should also work for Windows users.
I've tested this with Python 2.6, too.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #3737 from JoshRosen/update-boto and squashes the following commits:
0aa43cc [Josh Rosen] Remove unused setup_standalone_cluster() method.
f02935d [Josh Rosen] Enable Python deprecation warnings and fix one Boto warning:
587ae89 [Josh Rosen] [SPARK-4890] Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it
Diffstat (limited to 'ec2/spark-ec2')
| -rwxr-xr-x | ec2/spark-ec2 | 3 |
1 files changed, 1 insertions, 2 deletions
diff --git a/ec2/spark-ec2 b/ec2/spark-ec2 index 4aa908242e..3abd3f396f 100755 --- a/ec2/spark-ec2 +++ b/ec2/spark-ec2 @@ -22,5 +22,4 @@ #+ the underlying Python script. SPARK_EC2_DIR="$(dirname $0)" -PYTHONPATH="${SPARK_EC2_DIR}/third_party/boto-2.4.1.zip/boto-2.4.1:$PYTHONPATH" \ - python "${SPARK_EC2_DIR}/spark_ec2.py" "$@" +python -Wdefault "${SPARK_EC2_DIR}/spark_ec2.py" "$@" |