aboutsummaryrefslogtreecommitdiff
path: root/docs/running-on-yarn.md
Commit message (Collapse)AuthorAgeFilesLines
* SPARK-1408 Modify Spark on Yarn to point to the history server when app ...Thomas Graves2014-04-171-0/+1
| | | | | | | | | | | | | | | | ...finishes Note this is dependent on https://github.com/apache/spark/pull/204 to have a working history server, but there are no code dependencies. This also fixes SPARK-1288 yarn stable finishApplicationMaster incomplete. Since I was in there I made the diagnostic message be passed properly. Author: Thomas Graves <tgraves@apache.org> Closes #362 from tgravescs/SPARK-1408 and squashes the following commits: ec89705 [Thomas Graves] Fix typo. 446122d [Thomas Graves] Make config yarn specific f5d5373 [Thomas Graves] SPARK-1408 Modify Spark on Yarn to point to the history server when app finishes
* SPARK-1376. In the yarn-cluster submitter, rename "args" option to "arg"Sandy Ryza2014-04-011-3/+4
| | | | | | | | Author: Sandy Ryza <sandy@cloudera.com> Closes #279 from sryza/sandy-spark-1376 and squashes the following commits: d8aebfa [Sandy Ryza] SPARK-1376. In the yarn-cluster submitter, rename "args" option to "arg"
* SPARK-1126. spark-app preliminarySandy Ryza2014-03-291-2/+4
| | | | | | | | | | | | | | | | | This is a starting version of the spark-app script for running compiled binaries against Spark. It still needs tests and some polish. The only testing I've done so far has been using it to launch jobs in yarn-standalone mode against a pseudo-distributed cluster. This leaves out the changes required for launching python scripts. I think it might be best to save those for another JIRA/PR (while keeping to the design so that they won't require backwards-incompatible changes). Author: Sandy Ryza <sandy@cloudera.com> Closes #86 from sryza/sandy-spark-1126 and squashes the following commits: d428d85 [Sandy Ryza] Commenting, doc, and import fixes from Patrick's comments e7315c6 [Sandy Ryza] Fix failing tests 34de899 [Sandy Ryza] Change --more-jars to --jars and fix docs 299ddca [Sandy Ryza] Fix scalastyle a94c627 [Sandy Ryza] Add newline at end of SparkSubmit 04bc4e2 [Sandy Ryza] SPARK-1126. spark-submit script
* SPARK-1183. Don't use "worker" to mean executorSandy Ryza2014-03-131-15/+14
| | | | | | | | | | | | Author: Sandy Ryza <sandy@cloudera.com> Closes #120 from sryza/sandy-spark-1183 and squashes the following commits: 5066a4a [Sandy Ryza] Remove "worker" in a couple comments 0bd1e46 [Sandy Ryza] Remove --am-class from usage bfc8fe0 [Sandy Ryza] Remove am-class from doc and fix yarn-alpha 607539f [Sandy Ryza] Address review comments 74d087a [Sandy Ryza] SPARK-1183. Don't use "worker" to mean executor
* SPARK-1197. Change yarn-standalone to yarn-cluster and fix up running on ↵Sandy Ryza2014-03-061-29/+36
| | | | | | | | | | | | | YARN docs This patch changes "yarn-standalone" to "yarn-cluster" (but still supports the former). It also cleans up the Running on YARN docs and adds a section on how to view logs. Author: Sandy Ryza <sandy@cloudera.com> Closes #95 from sryza/sandy-spark-1197 and squashes the following commits: 563ef3a [Sandy Ryza] Review feedback 6ad06d4 [Sandy Ryza] Change yarn-standalone to yarn-cluster and fix up running on YARN docs
* SPARK-1053. Don't require SPARK_YARN_APP_JARSandy Ryza2014-02-261-4/+2
| | | | | | | | | | | | It looks this just requires taking out the checks. I verified that, with the patch, I was able to run spark-shell through yarn without setting the environment variable. Author: Sandy Ryza <sandy@cloudera.com> Closes #553 from sryza/sandy-spark-1053 and squashes the following commits: b037676 [Sandy Ryza] SPARK-1053. Don't require SPARK_YARN_APP_JAR
* [SPARK-1105] fix site scala version error in docsCodingCat2014-02-191-8/+8
| | | | | | | | | | | | | https://spark-project.atlassian.net/browse/SPARK-1105 fix site scala version error Author: CodingCat <zhunansjtu@gmail.com> Closes #618 from CodingCat/doc_version and squashes the following commits: 39bb8aa [CodingCat] more fixes 65bedb0 [CodingCat] fix site scala version error in doc
* Incorporate Tom's comments - update doc and code to reflect that core ↵Sandy Ryza2014-01-211-1/+1
| | | | requests may not always be honored
* SPARK-1033. Ask for cores in Yarn container requestsSandy Ryza2014-01-201-1/+1
|
* yarn-client addJar fix and misc otherThomas Graves2014-01-091-2/+13
|
* Merge pull request #345 from colorant/yarnThomas Graves2014-01-081-0/+2
|\ | | | | | | | | | | support distributing extra files to worker for yarn client mode So that user doesn't need to package all dependency into one assemble jar as spark app jar
| * Export --file for YarnClient mode to support sending extra files to worker ↵Raymond Liu2014-01-071-0/+2
| | | | | | | | on yarn cluster
* | Code review feedbackHolden Karau2014-01-051-3/+3
|/
* Merge remote-tracking branch 'apache-github/master' into remove-binariesPatrick Wendell2014-01-031-7/+4
|\ | | | | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/DriverSuite.scala docs/python-programming-guide.md
| * Merge pull request #317 from ScrapCodes/spark-915-segregate-scriptsPatrick Wendell2014-01-031-4/+4
| |\ | | | | | | | | | Spark-915 segregate scripts
| | * sbin/spark-class* -> bin/spark-class*Prashant Sharma2014-01-031-2/+2
| | |
| | * run-example -> bin/run-examplePrashant Sharma2014-01-021-1/+1
| | |
| | * spark-shell -> bin/spark-shellPrashant Sharma2014-01-021-1/+1
| | |
| | * Merge branch 'scripts-reorg' of github.com:shane-huang/incubator-spark into ↵Prashant Sharma2014-01-021-2/+2
| | |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | spark-915-segregate-scripts Conflicts: bin/spark-shell core/pom.xml core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala core/src/main/scala/org/apache/spark/ui/UIWorkloadGenerator.scala core/src/test/scala/org/apache/spark/DriverSuite.scala python/run-tests sbin/compute-classpath.sh sbin/spark-class sbin/stop-slaves.sh
| | | * added spark-class and spark-executor to sbinshane-huang2013-09-231-2/+2
| | | | | | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| * | | fix docs for yarnRaymond Liu2014-01-031-3/+0
| | | |
| * | | Update maven build documentationRaymond Liu2014-01-031-1/+1
| | | |
| * | | Fix yarn/README.md and update docs/running-on-yarn.mdRaymond Liu2014-01-031-1/+1
| |/ /
* | | fixed review commentsPrashant Sharma2014-01-031-2/+2
| | |
* | | Removed sbt folder and changed docs accordinglyPrashant Sharma2014-01-021-3/+3
|/ /
* | Fix list rendering in YARN markdown docs.Patrick Wendell2013-12-101-5/+7
| |
* | Various broken links in documentationPatrick Wendell2013-12-071-2/+2
| |
* | more docsAli Ghodsi2013-12-061-1/+1
| |
* | Updated documentation about the YARN v2.2 build processAli Ghodsi2013-12-061-0/+8
| |
* | Add YarnClientClusterScheduler and Backend.Raymond Liu2013-11-221-2/+25
| | | | | | | | | | | | | | With this scheduler, the user application is launched locally, While the executor will be launched by YARN on remote nodes. This enables spark-shell to run upon YARN.
* | Impove Spark on Yarn Error handlingtgravescs2013-11-191-0/+2
| |
* | Allow spark on yarn to be run from HDFS. Allows the spark.jar, app.jar, and ↵tgravescs2013-11-041-0/+1
| | | | | | | | log4j.properties to be put into hdfs.
* | Merge remote-tracking branch 'tgravescs/sparkYarnDistCache'Matei Zaharia2013-10-101-1/+8
|\ \ | | | | | | | | | | | | | | | | | | | | | Closes #11 Conflicts: docs/running-on-yarn.md yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
| * | Adding in the --addJars option to make SparkContext.addJar work on yarn and ↵tgravescs2013-10-031-0/+2
| | | | | | | | | | | | | | | | | | cleanup the classpaths
| * | Support distributed cache files and archives on spark on yarn and attempt to ↵Y.CORP.YAHOO.COM\tgraves2013-09-231-1/+6
| |/ | | | | | | cleanup the staging directory on exit
* / Allow users to set the application name for Spark on Yarntgravescs2013-10-021-0/+1
|/
* Clarify YARN exampleJey Kottalam2013-09-061-9/+22
|
* Review comment changes and update to org.apache packagingY.CORP.YAHOO.COM\tgraves2013-09-031-3/+3
|
* Merge remote-tracking branch 'mesos/master' into yarnUILinkY.CORP.YAHOO.COM\tgraves2013-09-031-28/+11
|\ | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/ui/UIUtils.scala core/src/main/scala/org/apache/spark/ui/jobs/PoolTable.scala core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala docs/running-on-yarn.md
| * More doc improvements + better warnings when you haven't built SparkMatei Zaharia2013-08-301-28/+11
| |
* | fix up minor thingsY.CORP.YAHOO.COM\tgraves2013-08-301-1/+1
| |
* | Link the Spark UI to the Yarn UIY.CORP.YAHOO.COM\tgraves2013-08-301-0/+4
|/
* Change build and run instructions to use assembliesMatei Zaharia2013-08-291-4/+4
| | | | | | | | | | | | | | | | This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
* Merge pull request #865 from tgravescs/fixtmpdirMatei Zaharia2013-08-281-3/+1
|\ | | | | Spark on Yarn should use yarn approved directories for spark.local.dir and tmp
| * Update docs and remove old reference to --user optionY.CORP.YAHOO.COM\tgraves2013-08-261-3/+1
| |
* | fix typosY.CORP.YAHOO.COM\tgraves2013-08-261-1/+1
| |
* | Add ability for user to specify environment variablesY.CORP.YAHOO.COM\tgraves2013-08-261-0/+6
|/
* Update build docsJey Kottalam2013-08-211-13/+7
|
* update YARN docsJey Kottalam2013-08-151-2/+2
|
* Fixed a couple typos and formating problems in the YARN documentation.Reynold Xin2013-05-171-9/+11
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-13 18:20:47 +0000