aboutsummaryrefslogtreecommitdiff
path: root/core/src/main/scala/org/apache
Commit message (Collapse)AuthorAgeFilesLines
* Changes based on review feedback.Patrick Wendell2014-01-067-22/+34
|
* Respect supervise option at MasterPatrick Wendell2013-12-291-3/+15
|
* Slight change to retry logicPatrick Wendell2013-12-291-2/+3
|
* TODO clean-upPatrick Wendell2013-12-295-6/+6
|
* Adding driver ID to submission responsePatrick Wendell2013-12-292-2/+2
|
* Documentation and adding supervise optionPatrick Wendell2013-12-294-13/+18
|
* Changes to allow fate sharing of drivers/executors and workers.Patrick Wendell2013-12-2916-127/+229
|
* Some notes and TODO about dependenciesPatrick Wendell2013-12-271-1/+7
|
* Minor fixesPatrick Wendell2013-12-263-20/+25
|
* Addressing smaller changes from Aaron's reviewPatrick Wendell2013-12-267-27/+31
|
* Merge pull request #1 from aarondav/driverPatrick Wendell2013-12-261-62/+31
|\ | | | | Refactor DriverClient to be more Actor-based
| * Refactor DriverClient to be more Actor-basedAaron Davidson2013-12-251-62/+31
| |
* | Removing accidental filePatrick Wendell2013-12-262-10/+1
| |
* | Updated approach to driver restartingPatrick Wendell2013-12-262-23/+30
|/
* Removing un-used variablePatrick Wendell2013-12-251-2/+0
|
* Small fix from rebasePatrick Wendell2013-12-251-1/+1
|
* Minor bug fixPatrick Wendell2013-12-252-1/+6
|
* Minor style clean-upPatrick Wendell2013-12-255-18/+16
|
* Import clean-up (yay Aaron)Patrick Wendell2013-12-2510-38/+33
|
* Adding scheduling and reporting based on coresPatrick Wendell2013-12-256-8/+14
|
* Adding better option parsingPatrick Wendell2013-12-259-42/+142
|
* Initial cut at driver submission.Patrick Wendell2013-12-2516-53/+781
|
* Renaming Client => AppClientPatrick Wendell2013-12-254-11/+12
|
* Merge pull request #127 from kayousterhout/consolidate_schedulersPatrick Wendell2013-12-2417-1202/+874
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Deduplicate Local and Cluster schedulers. The code in LocalScheduler/LocalTaskSetManager was nearly identical to the code in ClusterScheduler/ClusterTaskSetManager. The redundancy made making updating the schedulers unnecessarily painful and error- prone. This commit combines the two into a single TaskScheduler/ TaskSetManager. Unfortunately the diff makes this change look much more invasive than it is -- TaskScheduler.scala is only superficially changed (names updated, overrides removed) from the old ClusterScheduler.scala, and the same with TaskSetManager.scala. Thanks @rxin for suggesting this change!
| * Responded to Reynold's style commentsKay Ousterhout2013-12-243-6/+7
| |
| * Correctly merged in maxTaskFailures fixKay Ousterhout2013-12-221-1/+1
| |
| * Renamed ClusterScheduler to TaskSchedulerImplKay Ousterhout2013-12-2010-27/+27
| |
| * Merge remote branch 'upstream/master' into consolidate_schedulersKay Ousterhout2013-12-2098-604/+742
| |\ | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
| * \ Merge master into 127Aaron Davidson2013-12-0842-586/+1196
| |\ \
| * | | Fixed error message in ClusterScheduler to be consistent with the old ↵Kay Ousterhout2013-11-151-2/+6
| | | | | | | | | | | | | | | | LocalScheduler
| * | | Merge remote-tracking branch 'upstream/master' into consolidate_schedulersKay Ousterhout2013-11-156-79/+45
| |\ \ \ | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala
| * | | | Don't retry tasks if result wasn't serializableKay Ousterhout2013-11-141-1/+11
| | | | |
| * | | | Fix bug where scheduler could hang after task failure.Kay Ousterhout2013-11-141-10/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a task fails, we need to call reviveOffers() so that the task can be rescheduled on a different machine. In the current code, the state in ClusterTaskSetManager indicating which tasks are pending may be updated after revive offers is called (there's a race condition here), so when revive offers is called, the task set manager does not yet realize that there are failed tasks that need to be relaunched.
| * | | | Changed local backend to use Akka actorKay Ousterhout2013-11-141-23/+57
| | | | |
| * | | | Fixed naming issues and added back ability to specify max task failures.Kay Ousterhout2013-11-137-20/+80
| | | | |
| * | | | Merge remote-tracking branch 'upstream/master' into consolidate_schedulersKay Ousterhout2013-11-1329-504/+1455
| |\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/scheduler/ClusterScheduler.scala
| * | | | | Extracted TaskScheduler interface.Kay Ousterhout2013-11-1310-52/+57
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Also changed the default maximum number of task failures to be 0 when running in local mode.
| * | | | | Cleaned up imports and fixed test bugKay Ousterhout2013-10-312-3/+1
| | | | | |
| * | | | | Deduplicate Local and Cluster schedulers.Kay Ousterhout2013-10-3017-1675/+1247
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The code in LocalScheduler/LocalTaskSetManager was nearly identical to the code in ClusterScheduler/ClusterTaskSetManager. The redundancy made making updating the schedulers unnecessarily painful and error- prone. This commit combines the two into a single TaskScheduler/ TaskSetManager.
* | | | | | Merge pull request #279 from aarondav/shuffle-cleanup0Patrick Wendell2013-12-243-7/+35
|\ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Clean up shuffle files once their metadata is gone Previously, we would only clean the in-memory metadata for consolidated shuffle files. Additionally, fixes a bug where the Metadata Cleaner was ignoring type-specific TTLs.
| * | | | | | Clean up shuffle files once their metadata is goneAaron Davidson2013-12-193-7/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we would only clean the in-memory metadata for consolidated shuffle files. Additionally, fixes a bug where the Metadata Cleaner was ignoring type- specific TTLs.
* | | | | | | Merge pull request #277 from tdas/scheduler-updateMatei Zaharia2013-12-242-6/+2
|\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactored the streaming scheduler and added StreamingListener interface - Refactored the streaming scheduler for cleaner code. Specifically, the JobManager was renamed to JobScheduler, as it does the actual scheduling of Spark jobs to the SparkContext. The earlier Scheduler was renamed to JobGenerator, as it actually generates the jobs from the DStreams. The JobScheduler starts the JobGenerator. Also, moved all the scheduler related code from spark.streaming to spark.streaming.scheduler package. - Implemented the StreamingListener interface, similar to SparkListener. The streaming version of StatusReportListener prints the batch processing time statistics (for now). Added StreamingListernerSuite to test it. - Refactored streaming TestSuiteBase for deduping code in the other streaming testsuites.
| * | | | | | | Minor changes.Tathagata Das2013-12-181-1/+0
| | | | | | | |
| * | | | | | | Merge branch 'apache-master' into scheduler-updateTathagata Das2013-12-1892-525/+649
| |\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/dstream/ForEachDStream.scala
| * | | | | | | | Added StatsReportListener to generate processing time statistics across ↵Tathagata Das2013-12-181-4/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | multiple batches.
| * | | | | | | | Refactored streaming scheduler and added listener interface.Tathagata Das2013-12-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Refactored Scheduler + JobManager to JobGenerator + JobScheduler and added JobSet for cleaner code. Moved scheduler related code to streaming.scheduler package. - Added StreamingListener trait (similar to SparkListener) to enable gathering to streaming stats like processing times and delays. StreamingContext.addListener() to added listeners. - Deduped some code in streaming tests by modifying TestSuiteBase, and added StreamingListenerSuite.
* | | | | | | | | Merge pull request #244 from leftnoteasy/masterReynold Xin2013-12-236-5/+184
|\ \ \ \ \ \ \ \ \ | |_|_|_|_|_|_|_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added SPARK-968 implementation for review Added SPARK-968 implementation for review
| * | | | | | | | SPARK-968, added executor address showing in aggregated metrics by executors ↵wangda.tan2013-12-231-0/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | table
| * | | | | | | | added changes according to comments from rxinwangda.tan2013-12-227-44/+24
| | | | | | | | |
| * | | | | | | | spark-968, changes for avoid a NPEwangda.tan2013-12-172-25/+29
| | | | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-02 09:00:07 +0000