aboutsummaryrefslogtreecommitdiff
path: root/ec2
Commit message (Collapse)AuthorAgeFilesLines
...
* SPARK-1790: Update EC2 scripts to support r3 instance typesVarakhedi Sujeet2014-06-041-2/+12
| | | | | | | | Author: Varakhedi Sujeet <svarakhedi@gopivotal.com> Closes #960 from sujeetv/ec2-r3 and squashes the following commits: 3cb9fd5 [Varakhedi Sujeet] SPARK-1790: Update EC2 scripts to support r3 instance
* Update spark-ec2 scripts for 1.0.0 on masterAaron Davidson2014-06-031-3/+3
| | | | | | | | | | | | | The change was previously committed only to branch-1.0 as part of https://github.com/apache/spark/commit/a34e6fda1d6fb8e769c21db70845f1a6dde968d8 Author: Aaron Davidson <aaron@databricks.com> This patch had conflicts when merged, resolved by Committer: Patrick Wendell <pwendell@gmail.com> Closes #938 from aarondav/sparkec2 and squashes the following commits: 067cc31 [Aaron Davidson] Update spark-ec2 scripts for 1.0.0 on master
* Made spark_ec2.py PEP8 compliant.Reynold Xin2014-06-011-675/+712
| | | | | | | | | | | | The change set is actually pretty small -- mostly whitespace changes. Admittedly this is a scary change due to the lack of tests to cover the ec2 scripts, and also because indentation actually impacts control flow in Python ... Look at changes without whitespace diff here: https://github.com/apache/spark/pull/891/files?w=1 Author: Reynold Xin <rxin@apache.org> Closes #891 from rxin/spark-ec2-pep8 and squashes the following commits: ac1bf11 [Reynold Xin] Made spark_ec2.py PEP8 compliant.
* Version bump of spark-ec2 scriptsPatrick Wendell2014-05-161-1/+1
| | | | | | | | | | This will allow us to change things in spark-ec2 related to the 1.0 release. Author: Patrick Wendell <pwendell@gmail.com> Closes #809 from pwendell/spark-ec2 and squashes the following commits: 59117fb [Patrick Wendell] Version bump of spark-ec2 scripts
* Address SPARK-1717msiddalingaiah2014-05-041-3/+3
| | | | | | | | | | I tested the change locally with Spark 0.9.1, but I can't test with 1.0.0 because there was no AMI for it at the time. It's a trivial fix, so it shouldn't cause any problems. Author: msiddalingaiah <madhu@madhu.com> Closes #641 from msiddalingaiah/master and squashes the following commits: a4f7404 [msiddalingaiah] Address SPARK-1717
* EC2 script should exit with non-zero code on UsageErrorAllan Douglas R. de Oliveira2014-05-041-0/+1
| | | | | | | | | | This is specially import because some ssh errors are raised as UsageError, preventing an automated usage of the script from detecting the failure. Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com> Closes #638 from douglaz/ec2_exit_code_fix and squashes the following commits: 5915e6d [Allan Douglas R. de Oliveira] EC2 script should exit with non-zero code on UsageError
* EC2 configurable workersAllan Douglas R. de Oliveira2014-05-032-2/+12
| | | | | | | | | | | | | | Added option to configure number of worker instances and to set SPARK_MASTER_OPTS Depends on: https://github.com/mesos/spark-ec2/pull/46 Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com> Closes #612 from douglaz/ec2_configurable_workers and squashes the following commits: d6c5d65 [Allan Douglas R. de Oliveira] Added master opts parameter 6c34671 [Allan Douglas R. de Oliveira] Use number of worker instances as string on template ba528b9 [Allan Douglas R. de Oliveira] Added SPARK_WORKER_INSTANCES parameter
* Add Spark v0.9.1 to ec2 launch script and use it as the defaultHarvey Feng2014-04-101-2/+2
| | | | | | | | | | Mainly ported from branch-0.9. Author: Harvey Feng <hyfeng224@gmail.com> Closes #385 from harveyfeng/0.9.1-ec2 and squashes the following commits: 769ac2f [Harvey Feng] Add Spark v0.9.1 to ec2 launch script and use it as the default
* SPARK-1156: allow user to login into a cluster without slavesCodingCat2014-03-051-5/+6
| | | | | | | | | | | | | | Reported in https://spark-project.atlassian.net/browse/SPARK-1156 The current spark-ec2 script doesn't allow user to login to a cluster without slaves. One of the issues brought by this behaviour is that when all the worker died, the user cannot even login to the cluster for debugging, etc. Author: CodingCat <zhunansjtu@gmail.com> Closes #58 from CodingCat/SPARK-1156 and squashes the following commits: 104af07 [CodingCat] output ERROR to stderr 9a71769 [CodingCat] do not allow user to start 0-slave cluster 24a7c79 [CodingCat] allow user to login into a cluster without slaves
* Remove remaining references to incubationPatrick Wendell2014-03-022-2/+2
| | | | | | | | | | This removes some loose ends not caught by the other (incubating -> tlp) patches. @markhamstra this updates the version as you mentioned earlier. Author: Patrick Wendell <pwendell@gmail.com> Closes #51 from pwendell/tlp and squashes the following commits: d553b1b [Patrick Wendell] Remove remaining references to incubation
* SPARK-1106: check key name and identity file before launch a clusterXiangrui Meng2014-02-181-0/+6
| | | | | | | | | | | | I launched an EC2 cluster without providing a key name and an identity file. The error showed up after two minutes. It would be good to check those options before launch, given the fact that EC2 billing rounds up to hours. JIRA: https://spark-project.atlassian.net/browse/SPARK-1106 Author: Xiangrui Meng <meng@databricks.com> Closes #617 from mengxr/ec2 and squashes the following commits: 2dfb316 [Xiangrui Meng] check key name and identity file before launch a cluster
* Merge pull request #598 from shivaram/master.Shivaram Venkataraman2014-02-131-2/+2
| | | | | | | | | | | | Update spark_ec2 to use 0.9.0 by default Backports change from branch-0.9 Author: Shivaram Venkataraman <shivaram@eecs.berkeley.edu> Closes #598 and squashes the following commits: f6d3ed0 [Shivaram Venkataraman] Update spark_ec2 to use 0.9.0 by default Backports change from branch-0.9
* Add c3 instance types to Spark EC2Christian Lundgren2014-02-131-2/+12
| | | | | | | | | | | | | The number of disks for the c3 instance types taken from here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#StorageOnInstanceTypes Author: Christian Lundgren <christian.lundgren@gameanalytics.com> Closes #595 from chrisavl/branch-0.9 and squashes the following commits: c8af5f9 [Christian Lundgren] Add c3 instance types to Spark EC2 (cherry picked from commit 19b4bb2b444f1dbc4592bf3d58b17652e0ae6d6b) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* Add i2 instance types to Spark EC2.Shivaram Venkataraman2014-01-101-2/+10
|
* a few left over document changePrashant Sharma2014-01-021-1/+1
|
* Force pseudo-tty allocation in spark-ec2 script.Ewen Cheslack-Postava2013-12-161-2/+2
| | | | | | ssh commands need the -t argument repeated twice if there is no local tty, e.g. if the process running spark-ec2 uses nohup and the parent process exits.
* Enable stopping and starting a spot clusterAnkur Dave2013-11-111-19/+32
|
* expose UI port onlyHaoyuan Li2013-11-101-1/+1
|
* add tachyon moduleHaoyuan Li2013-11-091-7/+8
|
* update default githubFabrizio (Misto) Milo2013-11-011-1/+1
|
* Adding Shark 0.7.1 to EC2 scriptsPatrick Wendell2013-10-071-1/+1
|
* Bumping EC2 default version in master to `0.8.0`.Patrick Wendell2013-10-051-2/+2
|
* Update READMECruncherBigData2013-10-011-1/+1
|
* Merge pull request #670 from jey/ec2-ssh-improvementsReynold Xin2013-09-261-26/+80
|\ | | | | EC2 SSH improvements
| * Clarify error messages on SSH failureJey Kottalam2013-09-111-6/+21
| |
| * Generate new SSH key for the cluster, make "--identity-file" optionalJey Kottalam2013-09-061-21/+37
| |
| * Construct shell commands as sequences for safety and composabilityJey Kottalam2013-09-061-11/+34
| |
* | Fix issue with spark_ec2 seeing empty security groupsAaron Davidson2013-09-191-4/+4
| | | | | | | | | | | | | | | | | | | | Under unknown, but occasional, circumstances, reservation.groups is empty despite reservation.instances each having groups. This means that the spark_ec2 get_existing_clusters() method would fail to find any instances. To fix it, we simply use the instances' groups as the source of truth. Note that this is actually just a revival of PR #827, now that the issue has been reproduced.
* | Change port from 3030 to 4040Patrick Wendell2013-09-111-2/+1
| |
* | Adding Apache license to two filesPatrick Wendell2013-09-071-0/+17
|/
* Make initial connection failure message less daunting.Patrick Wendell2013-08-211-1/+1
| | | | | Right now it seems like something has gone wrong when this message is printed out. Instead, this is a normal condition. So I changed the message a bit.
* Merge branch 'master' into ec2-updatesPatrick Wendell2013-08-211-19/+17
|\ | | | | | | | | Conflicts: ec2/spark_ec2.py
| * Allow some wiggle room in UISuite port test and in EC2 portsMatei Zaharia2013-08-191-1/+1
| |
| * Small fixes to web UI:Matei Zaharia2013-08-191-1/+1
| | | | | | | | | | | | - Use SPARK_PUBLIC_DNS environment variable if set (for EC2) - Use a non-ephemeral port (3030 instead of 33000) by default - Updated test to use non-ephemeral port too
| * Open up Job UI ports (33000-33010) on EC2 clustersMatei Zaharia2013-07-291-19/+20
| |
* | Bump spark versionPatrick Wendell2013-08-201-2/+2
| |
* | Set default Hadoop version to 1Patrick Wendell2013-08-201-2/+2
| |
* | Adding web UI portPatrick Wendell2013-08-081-0/+1
| |
* | Point to new repository/branchPatrick Wendell2013-08-011-1/+1
| |
* | Pointing to new EC2 repositoryPatrick Wendell2013-08-011-3/+1
| |
* | Merge branch 'master' into ec2-updatesPatrick Wendell2013-07-313-2/+6
|\| | | | | | | | | Conflicts: ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh
| * Consistently invoke bash with /usr/bin/env bash in scripts to make code more ↵Ubuntu2013-07-182-2/+2
| | | | | | | | portable (JIRA Ticket SPARK-817)
| * Add Apache license headers and LICENSE and NOTICE filesMatei Zaharia2013-07-162-0/+4
| |
| * Fix SPARK-670: EC2 start command should require -i option.Josh Rosen2013-05-051-1/+1
| |
* | Don't install MapReduce for Hadoop1Patrick Wendell2013-07-301-0/+3
| |
* | Small improvement in python scriptPatrick Wendell2013-07-171-1/+1
| |
* | Bug fixPatrick Wendell2013-07-111-1/+1
| |
* | Removing support for custom Shark versionPatrick Wendell2013-07-111-11/+2
| |
* | Small fixPatrick Wendell2013-07-111-1/+1
| |
* | Adding ability to custom deploy only SparkPatrick Wendell2013-07-111-0/+7
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-13 13:04:24 +0000