aboutsummaryrefslogtreecommitdiff
path: root/yarn
Commit message (Collapse)AuthorAgeFilesLines
...
* SPARK-1211. In ApplicationMaster, set spark.master system property to "y...Sandy Ryza2014-03-102-0/+6
| | | | | | | | | | ...arn-cluster" Author: Sandy Ryza <sandy@cloudera.com> Closes #118 from sryza/sandy-spark-1211 and squashes the following commits: d4001c7 [Sandy Ryza] SPARK-1211. In ApplicationMaster, set spark.master system property to "yarn-cluster"
* SPARK-1193. Fix indentation in pom.xmlsSandy Ryza2014-03-072-8/+8
| | | | | | | | Author: Sandy Ryza <sandy@cloudera.com> Closes #91 from sryza/sandy-spark-1193 and squashes the following commits: a878124 [Sandy Ryza] SPARK-1193. Fix indentation in pom.xmls
* SPARK-1197. Change yarn-standalone to yarn-cluster and fix up running on ↵Sandy Ryza2014-03-061-1/+1
| | | | | | | | | | | | | YARN docs This patch changes "yarn-standalone" to "yarn-cluster" (but still supports the former). It also cleans up the Running on YARN docs and adds a section on how to view logs. Author: Sandy Ryza <sandy@cloudera.com> Closes #95 from sryza/sandy-spark-1197 and squashes the following commits: 563ef3a [Sandy Ryza] Review feedback 6ad06d4 [Sandy Ryza] Change yarn-standalone to yarn-cluster and fix up running on YARN docs
* SPARK-1189: Add Security to Spark - Akka, Http, ConnectionManager, UI use ↵Thomas Graves2014-03-066-33/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | | servlets resubmit pull request. was https://github.com/apache/incubator-spark/pull/332. Author: Thomas Graves <tgraves@apache.org> Closes #33 from tgravescs/security-branch-0.9-with-client-rebase and squashes the following commits: dfe3918 [Thomas Graves] Fix merge conflict since startUserClass now using runAsUser 05eebed [Thomas Graves] Fix dependency lost in upmerge d1040ec [Thomas Graves] Fix up various imports 05ff5e0 [Thomas Graves] Fix up imports after upmerging to master ac046b3 [Thomas Graves] Merge remote-tracking branch 'upstream/master' into security-branch-0.9-with-client-rebase 13733e1 [Thomas Graves] Pass securityManager and SparkConf around where we can. Switch to use sparkConf for reading config whereever possible. Added ConnectionManagerSuite unit tests. 4a57acc [Thomas Graves] Change UI createHandler routines to createServlet since they now return servlets 2f77147 [Thomas Graves] Rework from comments 50dd9f2 [Thomas Graves] fix header in SecurityManager ecbfb65 [Thomas Graves] Fix spacing and formatting b514bec [Thomas Graves] Fix reference to config ed3d1c1 [Thomas Graves] Add security.md 6f7ddf3 [Thomas Graves] Convert SaslClient and SaslServer to scala, change spark.authenticate.ui to spark.ui.acls.enable, and fix up various other things from review comments 2d9e23e [Thomas Graves] Merge remote-tracking branch 'upstream/master' into security-branch-0.9-with-client-rebase_rework 5721c5a [Thomas Graves] update AkkaUtilsSuite test for the actorSelection changes, fix typos based on comments, and remove extra lines I missed in rebase from AkkaUtils f351763 [Thomas Graves] Add Security to Spark - Akka, Http, ConnectionManager, UI to use servlets
* SPARK-1121: Include avro for yarn-alpha buildsPatrick Wendell2014-03-023-8/+37
| | | | | | | | | | | | | | | | | This lets us explicitly include Avro based on a profile for 0.23.X builds. It makes me sad how convoluted it is to express this logic in Maven. @tgraves and @sryza curious if this works for you. I'm also considering just reverting to how it was before. The only real problem was that Spark advertised a dependency on Avro even though it only really depends transitively on Avro through other deps. Author: Patrick Wendell <pwendell@gmail.com> Closes #49 from pwendell/avro-build-fix and squashes the following commits: 8d6ee92 [Patrick Wendell] SPARK-1121: Add avro to yarn-alpha profile
* Remove remaining references to incubationPatrick Wendell2014-03-023-3/+3
| | | | | | | | | | This removes some loose ends not caught by the other (incubating -> tlp) patches. @markhamstra this updates the version as you mentioned earlier. Author: Patrick Wendell <pwendell@gmail.com> Closes #51 from pwendell/tlp and squashes the following commits: d553b1b [Patrick Wendell] Remove remaining references to incubation
* SPARK-1051. On YARN, executors don't doAs submitting userSandy Ryza2014-02-284-2/+15
| | | | | | | | | | This reopens https://github.com/apache/incubator-spark/pull/538 against the new repo Author: Sandy Ryza <sandy@cloudera.com> Closes #29 from sryza/sandy-spark-1051 and squashes the following commits: 708ce49 [Sandy Ryza] SPARK-1051. doAs submitting user in YARN
* SPARK-1032. If Yarn app fails before registering, app master stays aroun...Sandy Ryza2014-02-282-18/+38
| | | | | | | | | | | | ...d long after This reopens https://github.com/apache/incubator-spark/pull/648 against the new repo. Author: Sandy Ryza <sandy@cloudera.com> Closes #28 from sryza/sandy-spark-1032 and squashes the following commits: 5953f50 [Sandy Ryza] SPARK-1032. If Yarn app fails before registering, app master stays around long after
* SPARK 1084.1 (resubmitted)Sean Owen2014-02-271-2/+2
| | | | | | | | | | | | | | | (Ported from https://github.com/apache/incubator-spark/pull/637 ) Author: Sean Owen <sowen@cloudera.com> Closes #31 from srowen/SPARK-1084.1 and squashes the following commits: 6c4a32c [Sean Owen] Suppress warnings about legitimate unchecked array creations, or change code to avoid it f35b833 [Sean Owen] Fix two misc javadoc problems 254e8ef [Sean Owen] Fix one new style error introduced in scaladoc warning commit 5b2fce2 [Sean Owen] Fix scaladoc invocation warning, and enable javac warnings properly, with plugin config updates 007762b [Sean Owen] Remove dead scaladoc links b8ff8cb [Sean Owen] Replace deprecated Ant <tasks> with <target>
* SPARK-1053. Don't require SPARK_YARN_APP_JARSandy Ryza2014-02-263-8/+5
| | | | | | | | | | | | It looks this just requires taking out the checks. I verified that, with the patch, I was able to run spark-shell through yarn without setting the environment variable. Author: Sandy Ryza <sandy@cloudera.com> Closes #553 from sryza/sandy-spark-1053 and squashes the following commits: b037676 [Sandy Ryza] SPARK-1053. Don't require SPARK_YARN_APP_JAR
* Merge pull request #542 from markhamstra/versionBump. Closes #542.Mark Hamstra2014-02-083-3/+3
| | | | | | | | | | | | | | | | | | Version number to 1.0.0-SNAPSHOT Since 0.9.0-incubating is done and out the door, we shouldn't be building 0.9.0-incubating-SNAPSHOT anymore. @pwendell Author: Mark Hamstra <markhamstra@gmail.com> == Merge branch commits == commit 1b00a8a7c1a7f251b4bb3774b84b9e64758eaa71 Author: Mark Hamstra <markhamstra@gmail.com> Date: Wed Feb 5 09:30:32 2014 -0800 Version number to 1.0.0-SNAPSHOT
* Merge pull request #526 from tgravescs/yarn_client_stop_am_fix. Closes #526.Thomas Graves2014-02-052-2/+2
| | | | | | | | | | | | | | | | | | | | spark on yarn - yarn-client mode doesn't always exit immediately https://spark-project.atlassian.net/browse/SPARK-1049 If you run in the yarn-client mode but you don't get all the workers you requested right away and then you exit your application, the application master stays around until it gets the number of workers you initially requested. This is a waste of resources. The AM should exit immediately upon the client going away. This fix simply checks to see if the driver closed while its waiting for the initial # of workers. Author: Thomas Graves <tgraves@apache.org> == Merge branch commits == commit 03f40a62584b6bdd094ba91670cd4aa6afe7cd81 Author: Thomas Graves <tgraves@apache.org> Date: Fri Jan 31 11:23:10 2014 -0600 spark on yarn - yarn-client mode doesn't always exit immediately
* Incorporate Tom's comments - update doc and code to reflect that core ↵Sandy Ryza2014-01-211-2/+1
| | | | requests may not always be honored
* SPARK-1033. Ask for cores in Yarn container requestsSandy Ryza2014-01-201-4/+5
|
* Address comments to fix code formatsRaymond Liu2014-01-144-24/+22
|
* Yarn workerRunnable refactorRaymond Liu2014-01-143-247/+184
|
* Yarn Client refactorRaymond Liu2014-01-145-709/+450
|
* Address code review concerns and comments.Henry Saputra2014-01-122-6/+6
|
* Merge branch 'master' into remove_simpleredundantreturn_scalaHenry Saputra2014-01-126-45/+88
|\
| * Update based on review commentsThomas Graves2014-01-094-27/+23
| |
| * add application typeThomas Graves2014-01-091-0/+1
| |
| * yarn-client addJar fix and misc otherThomas Graves2014-01-093-31/+77
| |
* | Remove simple redundant return statement for Scala methods/functions:Henry Saputra2014-01-126-20/+21
|/ | | | | | -) Only change simple return statements at the end of method -) Ignore the complex if-else check -) Ignore the ones inside synchronized
* Export --file for YarnClient mode to support sending extra files to worker ↵Raymond Liu2014-01-071-1/+3
| | | | on yarn cluster
* Minor typo fix for yarn clientRaymond Liu2014-01-072-2/+2
|
* merge upstream/masterliguoqiang2014-01-0322-73/+2251
|\
| * fix docs for yarnRaymond Liu2014-01-031-2/+2
| |
| * minor fix for loginfoRaymond Liu2014-01-031-1/+1
| |
| * move duplicate pom config into parent pomRaymond Liu2014-01-033-179/+84
| |
| * Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2Raymond Liu2014-01-0314-12/+12
| |
| * Fix yarn/README.mdRaymond Liu2014-01-031-6/+4
| |
| * Clean up unused files for yarnRaymond Liu2014-01-034-311/+0
| |
| * Fix pom for build yarn/2.x with yarn/common into one jarRaymond Liu2014-01-033-34/+200
| |
| * merge yarn/scheduler yarn/common code into one directoryRaymond Liu2014-01-033-0/+0
| |
| * Need to send dummy hello message to actually estabilish akka connection.Raymond Liu2014-01-032-0/+4
| |
| * A few clean up for yarn 2.0 codeRaymond Liu2014-01-032-8/+7
| |
| * Fix yarn/README.md and update docs/running-on-yarn.mdRaymond Liu2014-01-031-2/+0
| |
| * Add README for yarn modulesRaymond Liu2014-01-031-0/+16
| |
| * some code clean up for Yarn 2.2Raymond Liu2014-01-032-3/+3
| |
| * Fix pom file for scala binary versionRaymond Liu2014-01-036-8/+8
| |
| * Fix yarn/assemble pom fileRaymond Liu2014-01-032-0/+75
| |
| * Change profile name new-yarn to hadoop2.2-yarnRaymond Liu2014-01-032-2/+2
| |
| * Fix pom for yarn code reorgnaize commitRaymond Liu2014-01-036-489/+245
| |
| * Reorganize yarn related codes into sub projects to remove duplicate files.Raymond Liu2014-01-0321-18/+2580
| |
* | Modify spark on yarn to create SparkConf processliguoqiang2014-01-034-27/+36
| |
* | Modify spark on yarn to create SparkConf processliguoqiang2014-01-037-22/+27
|/
* fix yarn-clientThomas Graves2014-01-021-4/+5
|
* Fix yarn build after sparkConf changesThomas Graves2014-01-025-56/+48
|
* Merge remote-tracking branch 'origin/master' into conf2Matei Zaharia2013-12-295-11/+19
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala core/src/main/scala/org/apache/spark/scheduler/local/LocalScheduler.scala core/src/main/scala/org/apache/spark/util/MetadataCleaner.scala core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala new-yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala streaming/src/test/scala/org/apache/spark/streaming/WindowOperationsSuite.scala
| * Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarnliguoqiang2013-12-261-2/+1
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-23 20:31:04 +0000