aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #240 from pwendell/masterPatrick Wendell2013-12-071-4/+4
|\ | | | | | | SPARK-917 Improve API links in nav bar
| * SPARK-917 Improve API links in nav barPatrick Wendell2013-12-071-4/+4
| |
* | Merge pull request #239 from aarondav/nitPatrick Wendell2013-12-071-1/+1
|\ \ | |/ |/| | | Correct spellling error in configuration.md
| * Correct spellling error in configuration.mdAaron Davidson2013-12-071-1/+1
|/
* Merge pull request #237 from pwendell/formatting-fixPatrick Wendell2013-12-061-1/+0
|\ | | | | | | | | | | Formatting fix This is a single-line change. The diff appears larger here due to github being out of sync.
| * Minor formatting fix in config filePatrick Wendell2013-12-061-1/+0
|/
* Merge pull request #236 from pwendell/shuffle-docsPatrick Wendell2013-12-061-1/+1
|\ | | | | | | Adding disclaimer for shuffle file consolidation
| * Adding disclaimer for shuffle file consolidationPatrick Wendell2013-12-061-1/+1
| |
* | Merge pull request #235 from pwendell/masterPatrick Wendell2013-12-063-3/+10
|\ \ | |/ |/| | | Minor doc fixes and updating README
| * Minor doc fixes and updating READMEPatrick Wendell2013-12-063-3/+10
|/
* Merge pull request #234 from alig/masterPatrick Wendell2013-12-064-3/+17
|\ | | | | | | Updated documentation about the YARN v2.2 build process
| * more docsAli Ghodsi2013-12-063-3/+5
| |
| * Updated documentation about the YARN v2.2 build processAli Ghodsi2013-12-063-1/+13
| |
* | Merge pull request #190 from markhamstra/Stages4JobsMatei Zaharia2013-12-069-91/+280
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | stageId <--> jobId mapping in DAGScheduler Okay, I think this one is ready to go -- or at least it's ready for review and discussion. It's a carry-over of https://github.com/mesos/spark/pull/842 with updates for the newer job cancellation functionality. The prior discussion still applies. I've actually changed the job cancellation flow a bit: Instead of ``cancelTasks`` going to the TaskScheduler and then ``taskSetFailed`` coming back to the DAGScheduler (resulting in ``abortStage`` there), the DAGScheduler now takes care of figuring out which stages should be cancelled, tells the TaskScheduler to cancel tasks for those stages, then does the cleanup within the DAGScheduler directly without the need for any further prompting by the TaskScheduler. I know of three outstanding issues, each of which can and should, I believe, be handled in follow-up pull requests: 1) https://spark-project.atlassian.net/browse/SPARK-960 2) JobLogger should be re-factored to eliminate duplication 3) Related to 2), the WebUI should also become a consumer of the DAGScheduler's new understanding of the relationship between jobs and stages so that it can display progress indication and the like grouped by job. Right now, some of this information is just being sent out as part of ``SparkListenerJobStart`` messages, but more or different job <--> stage information may need to be exported from the DAGScheduler to meet listeners needs. Except for the eventQueue -> Actor commit, the rest can be cherry-picked almost cleanly into branch-0.8. A little merging is needed in MapOutputTracker and the DAGScheduler. Merged versions of those files are in https://github.com/markhamstra/incubator-spark/tree/aba2b40ce04ee9b7b9ea260abb6f09e050142d43 Note that between the recent Actor change in the DAGScheduler and the cleaning up of DAGScheduler data structures on job completion in this PR, some races have been introduced into the DAGSchedulerSuite. Those tests usually pass, and I don't think that better-behaved code that doesn't directly inspect DAGScheduler data structures should be seeing any problems, but I'll work on fixing DAGSchedulerSuite as either an addition to this PR or as a separate request. UPDATE: Fixed the race that I introduced. Created a JIRA issue (SPARK-965) for the one that was introduced with the switch to eventProcessorActor in the DAGScheduler.
| * | SparkListenerJobStart posted from local jobsMark Hamstra2013-12-031-0/+1
| | |
| * | Synchronous, inline cleanup after runLocallyMark Hamstra2013-12-033-13/+6
| | |
| * | Local jobs post SparkListenerJobEnd, and DAGScheduler data structureMark Hamstra2013-12-032-8/+11
| | | | | | | | | | | | cleanup always occurs before any posting of SparkListenerJobEnd.
| * | Tightly couple stageIdToJobIds and jobIdToStageIdsMark Hamstra2013-12-031-17/+12
| | |
| * | Cleaned up job cancellation handlingMark Hamstra2013-12-031-7/+5
| | |
| * | Refactoring to make job removal, stage removal, task cancellation clearerMark Hamstra2013-12-031-37/+39
| | |
| * | Improved commentMark Hamstra2013-12-031-4/+3
| | |
| * | Removed redundant residual re: reverted refactoring.Mark Hamstra2013-12-031-1/+1
| | |
| * | Fixed intended side-effectsMark Hamstra2013-12-031-2/+2
| | |
| * | Actor instead of eventQueue for LocalJobCompletedMark Hamstra2013-12-031-1/+1
| | |
| * | Added stageId <--> jobId mapping in DAGSchedulerMark Hamstra2013-12-039-88/+286
| | | | | | | | | | | | | | | ...and make sure that DAGScheduler data structures are cleaned up on job completion. Initial effort and discussion at https://github.com/mesos/spark/pull/842
* | | Merge pull request #233 from hsaputra/changecontexttobackendMatei Zaharia2013-12-061-2/+2
|\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Change the name of input argument in ClusterScheduler#initialize from context to backend. The SchedulerBackend used to be called ClusterSchedulerContext so just want to make small change of the input param in the ClusterScheduler#initialize to reflect this.
| * | | Change the name of input ragument in ClusterScheduler#initialize from ↵Henry Saputra2013-12-051-2/+2
| | |/ | |/| | | | | | | | | | | | | | | | context to backend. The SchedulerBackend used to be called ClusterSchedulerContext so just want to make small change of the input param in the ClusterScheduler#initialize to reflect this.
* | | Merge pull request #205 from kayousterhout/loggingMatei Zaharia2013-12-061-2/+34
|\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added logging of scheduler delays to UI This commit adds two metrics to the UI: 1) The time to get task results, if they're fetched remotely 2) The scheduler delay. When the scheduler starts getting overwhelmed (because it can't keep up with the rate at which tasks are being submitted), the result is that tasks get delayed on the tail-end: the message from the worker saying that the task has completed ends up in a long queue and takes a while to be processed by the scheduler. This commit records that delay in the UI so that users can tell when the scheduler is becoming the bottleneck.
| * | | Fixed problem with scheduler delayKay Ousterhout2013-12-021-4/+7
| | | |
| * | | Added logging of scheduler delays to UIKay Ousterhout2013-11-211-2/+31
| | | |
* | | | Merge pull request #220 from rxin/zippartMatei Zaharia2013-12-061-16/+11
|\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Memoize preferred locations in ZippedPartitionsBaseRDD so preferred location computation doesn't lead to exponential explosion. This was a problem in GraphX where we have a whole chain of RDDs that are ZippedPartitionsRDD's, and the preferred locations were taking eternity to compute. (cherry picked from commit e36fe55a031d2c01c9d7c5d85965951c681a0c74) Signed-off-by: Reynold Xin <rxin@apache.org>
| * | | | Memoize preferred locations in ZippedPartitionsBaseRDD so preferred location ↵Reynold Xin2013-11-301-16/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | computation doesn't lead to exponential explosion. (cherry picked from commit e36fe55a031d2c01c9d7c5d85965951c681a0c74) Signed-off-by: Reynold Xin <rxin@apache.org>
* | | | | Merge pull request #232 from markhamstra/FiniteWaitReynold Xin2013-12-053-1/+28
|\ \ \ \ \ | |_|_|/ / |/| | | | | | | | | | | | | | | | | | | jobWaiter.synchronized before jobWaiter.wait ...else ``IllegalMonitorStateException`` in ``SimpleFutureAction#ready``.
| * | | | FutureAction result testsMark Hamstra2013-12-051-0/+26
| | | | |
| * | | | jobWaiter.synchronized before jobWaiter.waitMark Hamstra2013-12-052-1/+2
|/ / / /
* | | | Merge pull request #228 from pwendell/masterPatrick Wendell2013-12-053-4/+49
|\ \ \ \ | | | | | | | | | | | | | | | Document missing configs and set shuffle consolidation to false.
| * | | | Forcing shuffle consolidation in DiskBlockManagerSuitePatrick Wendell2013-12-051-2/+12
| | | | |
| * | | | Small changes from Matei reviewPatrick Wendell2013-12-041-2/+2
| | | | |
| * | | | Document missing configs and set shuffle consolidation to false.Patrick Wendell2013-12-042-2/+37
| | | | |
* | | | | Merge pull request #199 from harveyfeng/yarn-2.2Matei Zaharia2013-12-0423-343/+3716
|\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Hadoop 2.2 migration Includes support for the YARN API stabilized in the Hadoop 2.2 release, and a few style patches. Short description for each set of commits: a98f5a0 - "Misc style changes in the 'yarn' package" a67ebf4 - "A few more style fixes in the 'yarn' package" Both of these are some minor style changes, such as fixing lines over 100 chars, to the existing YARN code. ab8652f - "Add a 'new-yarn' directory ... " Copies everything from `SPARK_HOME/yarn` to `SPARK_HOME/new-yarn`. No actual code changes here. 4f1c3fa - "Hadoop 2.2 YARN API migration ..." API patches to code in the `SPARK_HOME/new-yarn` directory. There are a few more small style changes mixed in, too. Based on @colorant's Hadoop 2.2 support for the scala-2.10 branch in #141. a1a1c62 - "Add optional Hadoop 2.2 settings in sbt build ... " If Spark should be built against Hadoop 2.2, then: a) the `org.apache.spark.deploy.yarn` package will be compiled from the `new-yarn` directory. b) Protobuf v2.5 will be used as a Spark dependency, since Hadoop 2.2 depends on it. Also, Spark will be built against a version of Akka v2.0.5 that's built against Protobuf 2.5, named `akka-2.0.5-protobuf-2.5`. The patched Akka is here: https://github.com/harveyfeng/akka/tree/2.0.5-protobuf-2.5, and was published to local Ivy during testing. There's also a new boolean environment variable, `SPARK_IS_NEW_HADOOP`, that users can manually set if their `SPARK_HADOOP_VERSION` specification does not start with `2.2`, which is how the build file tries to detect a 2.2 version. Not sure if this is necessary or done in the best way, though...
| * \ \ \ \ Merge pull request #2 from colorant/yarn-client-2.2Harvey Feng2013-12-033-24/+56
| |\ \ \ \ \ | | | | | | | | | | | | | | Fix pom.xml for maven build
| | * | | | | Fix pom.xml for maven buildRaymond Liu2013-12-033-24/+56
| |/ / / / /
| * | | | | Use published "org.spark-project.akka-*" in sbt build for Hadoop-2.2 ↵Harvey Feng2013-12-031-13/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | dependencies. This also includes: -Change `isNewYarn` to `isNewHadoop`, since the protobuf-2.5 dependency is from Hadoop-2.2 itself. -Regexp bugix Credits to @alig for this patch.
| * | | | | Merge pull request #1 from colorant/yarn-client-2.2Harvey Feng2013-11-275-17/+405
| |\ \ \ \ \ | | | | | | | | | | | | | | Port yarn-client mode for new-yarn
| | * | | | | Port yarn-client mode for new-yarnRaymond Liu2013-11-275-17/+405
| |/ / / / /
| * | | | | Merge remote-tracking branch 'origin/master' into yarn-2.2Harvey Feng2013-11-2635-218/+1466
| |\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
| * | | | | | Add optional Hadoop 2.2 settings in sbt build.Harvey Feng2013-11-261-9/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If the Hadoop used is version 2.2 or derived from it, then Spark will be compiled against protobuf-2.5 and a protobuf-2.5 version of Akka 2.0.5.
| * | | | | | Hadoop 2.2 YARN API migration for `SPARK_HOME/new-yarn`Harvey Feng2013-11-236-489/+468
| | | | | | |
| * | | | | | Add a "new-yarn" directory in SPARK_HOME, intended to contain Hadoop-2.2 API ↵Harvey Feng2013-11-2311-0/+2822
| | | | | | | | | | | | | | | | | | | | | | | | | | | | changes.
| * | | | | | A few more style fixes in `yarn` package.Harvey Feng2013-11-233-45/+71
| | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-08 10:03:30 +0000