aboutsummaryrefslogtreecommitdiff
path: root/docs/ec2-scripts.md
Commit message (Collapse)AuthorAgeFilesLines
* [SPARK-14742][DOCS] Redirect spark-ec2 doc to new locationSean Owen2016-04-201-0/+7
| | | | | | | | | | | | | | ## What changes were proposed in this pull request? Restore `ec2-scripts.md` as a redirect to amplab/spark-ec2 docs ## How was this patch tested? `jekyll build` and checked with the browser Author: Sean Owen <sowen@cloudera.com> Closes #12534 from srowen/SPARK-14742.
* [SPARK-12735] Consolidate & move spark-ec2 to AMPLab managed repository.Reynold Xin2016-01-091-192/+0
| | | | | | Author: Reynold Xin <rxin@databricks.com> Closes #10673 from rxin/SPARK-12735.
* [SPARK-6402][DOC] - Remove some refererences to shark in docs and ec2Pierre Borckmans2015-03-191-1/+1
| | | | | | | | | | | | | | | EC2 script and job scheduling documentation still refered to Shark. I removed these references. I also removed a remaining `SHARK_VERSION` variable from `ec2-variables.sh`. Author: Pierre Borckmans <pierre.borckmans@realimpactanalytics.com> Closes #5083 from pierre-borckmans/remove_refererences_to_shark_in_docs and squashes the following commits: 4e90ffc [Pierre Borckmans] Removed deprecated SHARK_VERSION caea407 [Pierre Borckmans] Remove shark reference from ec2 script doc 196c744 [Pierre Borckmans] Removed references to Shark
* Update ec2-scripts.mdMiguel Peralvo2015-02-061-1/+1
| | | | | | | | | | | Change spark-version from 1.1.0 to 1.2.0 in the example for spark-ec2/Launch Cluster. Author: Miguel Peralvo <miguel.peralvo@gmail.com> Closes #4300 from MiguelPeralvo/patch-1 and squashes the following commits: 38adf0b [Miguel Peralvo] Update ec2-scripts.md 1850869 [Miguel Peralvo] Update ec2-scripts.md
* [SPARK-3405] add subnet-id and vpc-id options to spark_ec2.pyMike Jennings2014-12-161-0/+19
| | | | | | | | | | | | | | | | | | | | Based on this gist: https://gist.github.com/amar-analytx/0b62543621e1f246c0a2 We use security group ids instead of security group to get around this issue: https://github.com/boto/boto/issues/350 Author: Mike Jennings <mvj101@gmail.com> Author: Mike Jennings <mvj@google.com> Closes #2872 from mvj101/SPARK-3405 and squashes the following commits: be9cb43 [Mike Jennings] `pep8 spark_ec2.py` runs cleanly. 4dc6756 [Mike Jennings] Remove duplicate comment 731d94c [Mike Jennings] Update for code review. ad90a36 [Mike Jennings] Merge branch 'master' of https://github.com/apache/spark into SPARK-3405 1ebffa1 [Mike Jennings] Merge branch 'master' into SPARK-3405 52aaeec [Mike Jennings] [SPARK-3405] add subnet-id and vpc-id options to spark_ec2.py
* [SPARK-4652][DOCS] Add docs about spark-git-repo optionlewuathe2014-12-041-0/+5
| | | | | | | | | | | | | | | | There might be some cases when WIPS spark version need to be run on EC2 cluster. In order to setup this type of cluster more easily, add --spark-git-repo option description to ec2 documentation. Author: lewuathe <lewuathe@me.com> Author: Josh Rosen <joshrosen@databricks.com> Closes #3513 from Lewuathe/doc-for-development-spark-cluster and squashes the following commits: 6dae8ee [lewuathe] Wrap consistent with other descriptions cfaf9be [lewuathe] Add docs about spark-git-repo option (Editing / cleanup by Josh Rosen)
* [Spark-4509] Revert EC2 tag-based cluster membership patchXiangrui Meng2014-11-251-8/+6
| | | | | | | | | | | | | | | | | | | | | | This PR reverts changes related to tag-based cluster membership. As discussed in SPARK-3332, we didn't figure out a safe strategy to use tags to determine cluster membership, because tagging is not atomic. The following changes are reverted: SPARK-2333: 94053a7b766788bb62e2dbbf352ccbcc75f71fc0 SPARK-3213: 7faf755ae4f0cf510048e432340260a6e609066d SPARK-3608: 78d4220fa0bf2f9ee663e34bbf3544a5313b02f0. I tested launch, login, and destroy. It is easy to check the diff by comparing it to Josh's patch for branch-1.1: https://github.com/apache/spark/pull/2225/files JoshRosen I sent the PR to master. It might be easier for us to keep master and branch-1.2 the same at this time. We can always re-apply the patch once we figure out a stable solution. Author: Xiangrui Meng <meng@databricks.com> Closes #3453 from mengxr/SPARK-4509 and squashes the following commits: f0b708b [Xiangrui Meng] revert 94053a7b766788bb62e2dbbf352ccbcc75f71fc0 4298ea5 [Xiangrui Meng] revert 7faf755ae4f0cf510048e432340260a6e609066d 35963a1 [Xiangrui Meng] Revert "SPARK-3608 Break if the instance tag naming succeeds"
* stop, start and destroy require the EC2_REGIONJeff Steinmetz2014-09-261-10/+19
| | | | | | | | | | | | | | | | | i.e ./spark-ec2 --region=us-west-1 stop yourclustername Author: Jeff Steinmetz <jeffrey.steinmetz@gmail.com> Closes #2473 from jeffsteinmetz/master and squashes the following commits: 7491f2c [Jeff Steinmetz] fix case in EC2 cluster setup documentation bd3d777 [Jeff Steinmetz] standardized ec2 documenation to use <lower-case> sample args 2bf4a57 [Jeff Steinmetz] standardized ec2 documenation to use <lower-case> sample args 68d8372 [Jeff Steinmetz] standardized ec2 documenation to use <lower-case> sample args d2ab6e2 [Jeff Steinmetz] standardized ec2 documenation to use <lower-case> sample args 520e6dc [Jeff Steinmetz] standardized ec2 documenation to use <lower-case> sample args 37fc876 [Jeff Steinmetz] stop, start and destroy require the EC2_REGION
* [SPARK-787] Add S3 configuration parameters to the EC2 deploy scriptsDan Osipov2014-09-161-1/+1
| | | | | | | | | | | | | | | When deploying to AWS, there is additional configuration that is required to read S3 files. EMR creates it automatically, there is no reason that the Spark EC2 script shouldn't. This PR requires a corresponding PR to the mesos/spark-ec2 to be merged, as it gets cloned in the process of setting up machines: https://github.com/mesos/spark-ec2/pull/58 Author: Dan Osipov <daniil.osipov@shazam.com> Closes #1120 from danosipov/s3_credentials and squashes the following commits: 758da8b [Dan Osipov] Modify documentation to include the new parameter 71fab14 [Dan Osipov] Use a parameter --copy-aws-credentials to enable S3 credential deployment 7e0da26 [Dan Osipov] Get AWS credentials out of boto connection instance 39bdf30 [Dan Osipov] Add S3 configuration parameters to the EC2 deploy scripts
* SPARK-2333 - spark_ec2 script should allow option for existing security groupVida Ha2014-08-191-6/+8
| | | | | | | | | | | | | - Uses the name tag to identify machines in a cluster. - Allows overriding the security group name so it doesn't need to coincide with the cluster name. - Outputs the request id's of up to 10 pending spot instance requests. Author: Vida Ha <vida@databricks.com> Closes #1899 from vidaha/vida/ec2-reuse-security-group and squashes the following commits: c80d5c3 [Vida Ha] wrap retries in a try catch block b2989d5 [Vida Ha] SPARK-2333: spark_ec2 script should allow option for existing security group
* fix persistent-hdfsFabrizio (Misto) Milo2013-11-011-1/+1
|
* More fair scheduler docs and property names.Matei Zaharia2013-09-081-4/+4
| | | | | Also changed uses of "job" terminology to "application" when they referred to an entire Spark program, to avoid confusion.
* Version bump for ec2 docsPatrick Wendell2013-08-241-1/+1
|
* Merge branch 'master' into ec2-updatesPatrick Wendell2013-07-311-3/+2
|\ | | | | | | | | Conflicts: ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh
| * Made use of spark.executor.memory setting consistent and documented itMatei Zaharia2013-06-301-3/+2
| | | | | | | | | | | | Conflicts: core/src/main/scala/spark/SparkContext.scala
* | Updating relevant documentationPatrick Wendell2013-07-311-18/+14
|/
* Some tweaks to docsMatei Zaharia2013-02-261-7/+10
|
* Changes based on Matei's commentPatrick Wendell2013-01-201-2/+3
|
* Clarifying log directory in EC2 guidePatrick Wendell2013-01-191-1/+2
|
* Adds liquid variables to docs templating system so that they can be usedAndy Konwinski2012-10-081-2/+2
| | | | | | | | | throughout the docs: SPARK_VERSION, SCALA_VERSION, and MESOS_VERSION. To use them, e.g. use {{site.SPARK_VERSION}}. Also removes uses of {{HOME_PATH}} which were being resolved to "" by the templating system anyway.
* Updates to standalone cluster, web UI and deploy docs.Matei Zaharia2012-09-261-7/+6
|
* More updates to docs, including tuning guideMatei Zaharia2012-09-261-0/+10
|
* - Add docs/api to .gitignoreAndy Konwinski2012-09-161-15/+19
| | | | | | | | | | | | | - Rework/expand the nav bar with more of the docs site - Removing parts of docs about EC2 and Mesos that differentiate between running 0.5 and before - Merged subheadings from running-on-amazon-ec2.html that are still relevant (i.e., "Using a newer version of Spark" and "Accessing Data in S3") into ec2-scripts.html and deleted running-on-amazon-ec2.html - Added some TODO comments to a few docs - Updated the blurb about AMP Camp - Renamed programming-guide to spark-programming-guide - Fixing typos/etc. in Standalone Spark doc
* Adds ec2-scripts.md back (it was mistakenly removed earlier due toAndy Konwinski2012-09-121-0/+146
| | | | git weirdness).
* Small tweaks to generated doc pagesMatei Zaharia2012-09-121-146/+0
|
* Fixing a hanging sentence in docs/ec2-scripts.mdAndy Konwinski2012-09-121-1/+1
|
* Fixing lots of broken links.Andy Konwinski2012-09-121-7/+7
|
* Updated base README to point to documentation site instead of wiki, updatedAndy Konwinski2012-09-121-0/+146
docs/README.md to describe use of Jekyll, and renmaed things to make them more consistent with the lower-case-with-hyphens convention.
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-09 15:35:02 +0000