aboutsummaryrefslogtreecommitdiff
path: root/core
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #74 from rxin/killMatei Zaharia2013-10-184-7/+75
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Job cancellation via job group id. This PR adds a simple API to group together a set of jobs belonging to a thread and threads spawned from it. It also allows the cancellation of all jobs in this group. An example: sc.setJobDescription("this_is_the_group_id", "some job description") sc.parallelize(1 to 10000, 2).map { i => Thread.sleep(10); i }.count() In a separate thread: sc.cancelJobGroup("this_is_the_group_id")
| * Job cancellation via job group id.Reynold Xin2013-10-184-7/+75
| |
* | Merge pull request #68 from mosharaf/masterMatei Zaharia2013-10-185-9/+308
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Faster and stable/reliable broadcast HttpBroadcast is noticeably slow, but the alternatives (TreeBroadcast or BitTorrentBroadcast) are notoriously unreliable. The main problem with them is they try to manage the memory for the pieces of a broadcast themselves. Right now, the BroadcastManager does not know which machines the tasks reading from a broadcast variable is running and when they have finished. Consequently, we try to guess and often guess wrong, which blows up the memory usage and kills/hangs jobs. This very simple implementation solves the problem by not trying to manage the intermediate pieces; instead, it offloads that duty to the BlockManager which is quite good at juggling blocks. Otherwise, it is very similar to the BitTorrentBroadcast implementation (without fancy optimizations). And it runs much faster than HttpBroadcast we have right now. I've been using this for another project for last couple of weeks, and just today did some benchmarking against the Http one. The following shows the improvements for increasing broadcast size for cold runs. Each line represent the number of receivers. ![fix-bc-first](https://f.cloud.github.com/assets/232966/1349342/ffa149e4-36e7-11e3-9fa6-c74555829356.png) After the first broadcast is over, i.e., after JVM is wormed up and for HttpBroadcast the server is already running (I think), the following are the improvements for warm runs. ![fix-bc-succ](https://f.cloud.github.com/assets/232966/1349352/5a948bae-36e8-11e3-98ce-34f19ebd33e0.jpg) The curves are not as nice as the cold runs, but the improvements are obvious, specially for larger broadcasts and more receivers. Depending on how it goes, we should deprecate and/or remove old TreeBroadcast and BitTorrentBroadcast implementations, and hopefully, SPARK-889 will not be necessary any more.
| * | Should compile now.Mosharaf Chowdhury2013-10-171-1/+2
| | |
| * | Added an after block to reset spark.broadcast.factoryMosharaf Chowdhury2013-10-171-0/+4
| | |
| * | BroadcastSuite updated to test both HttpBroadcast and TorrentBroadcast in ↵Mosharaf Chowdhury2013-10-171-3/+44
| | | | | | | | | | | | local, local[N], local-cluster settings.
| * | Merge remote-tracking branch 'upstream/master'Mosharaf Chowdhury2013-10-179-105/+63
| |\|
| * | Code styling. Updated doc.Mosharaf Chowdhury2013-10-171-4/+4
| | |
| * | Removed unused code. Mosharaf Chowdhury2013-10-172-14/+11
| | | | | | | | | Changes to match Spark coding style.
| * | Fixes for the new BlockId naming convention.Mosharaf Chowdhury2013-10-162-7/+14
| | |
| * | Default blockSize is 4MB.Mosharaf Chowdhury2013-10-161-1/+1
| | | | | | | | | | | | BroadcastTest2 example added for testing broadcasts.
| * | Removed unnecessary code, and added comment of memory-latency tradeoff.Mosharaf Chowdhury2013-10-161-4/+6
| | |
| * | Torrent-ish broadcast based on BlockManager.Mosharaf Chowdhury2013-10-163-4/+251
| | |
* | | Spark shell exits if it cannot create SparkContextAaron Davidson2013-10-171-7/+6
| |/ |/| | | | | | | | | Mainly, this occurs if you provide a messed up MASTER url (one that doesn't match one of our regexes). Previously, we would default to Mesos, fail, and then start the shell anyway, except that any Spark command would fail.
* | Fixed unit testsKay Ousterhout2013-10-162-25/+26
| |
* | Removed TaskSchedulerListener interface.Kay Ousterhout2013-10-167-80/+37
|/ | | | | | | | | The interface was used only by the DAG scheduler (so it wasn't necessary to define the additional interface), and the naming makes it very confusing when reading the code (because "listener" was used to describe the DAG scheduler, rather than SparkListeners, which implement a nearly-identical interface but serve a different function).
* Merge pull request #62 from harveyfeng/masterMatei Zaharia2013-10-152-2/+5
|\ | | | | | | Make TaskContext's stageId publicly accessible.
| * Proper formatting for SparkHadoopWriter class extensions.Harvey Feng2013-10-151-1/+3
| |
| * Fix line length > 100 chars in SparkHadoopWriterHarvey Feng2013-10-151-1/+2
| |
| * Make TaskContext's stageId publicly accessible.Harvey Feng2013-10-151-1/+1
| |
* | Merge pull request #34 from kayousterhout/renameMatei Zaharia2013-10-156-36/+42
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Renamed StandaloneX to CoarseGrainedX. (as suggested by @rxin here https://github.com/apache/incubator-spark/pull/14) The previous names were confusing because the components weren't just used in Standalone mode. The scheduler used for Standalone mode is called SparkDeploySchedulerBackend, so referring to the base class as StandaloneSchedulerBackend was misleading.
| * | Fixed build error after merging in masterKay Ousterhout2013-10-151-1/+1
| | |
| * | Merge remote branch 'upstream/master' into renameKay Ousterhout2013-10-15132-1205/+4531
| |\|
| * | Added back fully qualified class nameKay Ousterhout2013-10-061-1/+1
| | |
| * | Renamed StandaloneX to CoarseGrainedX.Kay Ousterhout2013-10-046-35/+41
| | | | | | | | | | | | | | | | | | | | | The previous names were confusing because the components weren't just used in Standalone mode -- in fact, the scheduler used for Standalone mode is called SparkDeploySchedulerBackend. So, the previous names were misleading.
* | | Unified daemon thread poolsKay Ousterhout2013-10-157-38/+29
| |/ |/|
* | Bump up logging level to warning for failed tasks.Reynold Xin2013-10-141-5/+5
| |
* | Merge branch 'master' of github.com:apache/incubator-spark into killReynold Xin2013-10-1448-439/+635
|\ \ | | | | | | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
| * \ Merge pull request #57 from aarondav/bidReynold Xin2013-10-1440-369/+527
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactor BlockId into an actual type Converts all of our BlockId strings into actual BlockId types. Here are some advantages of doing this now: + Type safety + Code clarity - it's now obvious what the key of a shuffle or rdd block is, for instance. Additionally, appearing in tuple/map type signatures is a big readability bonus. A Seq[(String, BlockStatus)] is not very clear. Further, we can now use more Scala features, like matching on BlockId types. + Explicit usage - we can now formally tell where various BlockIds are being used (without doing string searches); this makes updating current BlockIds a much clearer process, and compiler-supported. (I'm looking at you, shuffle file consolidation.) + It will only get harder to make this change as time goes on. Downside is, of course, that this is a very invasive change touching a lot of different files, which will inevitably lead to merge conflicts for many.
| | * | Address Matei's commentsAaron Davidson2013-10-148-34/+28
| | | |
| | * | Change BlockId filename to name + rest of Patrick's commentsAaron Davidson2013-10-1311-36/+39
| | | |
| | * | Add unit test and address rest of Reynold's commentsAaron Davidson2013-10-1210-20/+144
| | | |
| | * | Refactor BlockId into an actual typeAaron Davidson2013-10-1239-369/+406
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is an unfortunately invasive change which converts all of our BlockId strings into actual BlockId types. Here are some advantages of doing this now: + Type safety + Code clarity - it's now obvious what the key of a shuffle or rdd block is, for instance. Additionally, appearing in tuple/map type signatures is a big readability bonus. A Seq[(String, BlockStatus)] is not very clear. Further, we can now use more Scala features, like matching on BlockId types. + Explicit usage - we can now formally tell where various BlockIds are being used (without doing string searches); this makes updating current BlockIds a much clearer process, and compiler-supported. (I'm looking at you, shuffle file consolidation.) + It will only get harder to make this change as time goes on. Since this touches a lot of files, it'd be best to either get this patch in quickly or throw it on the ground to avoid too many secondary merge conflicts.
| * | | Merge pull request #52 from harveyfeng/hadoop-closureReynold Xin2013-10-122-55/+26
| |\ \ \ | | |/ / | |/| | | | | | | | | | | | | | Add an optional closure parameter to HadoopRDD instantiation to use when creating local JobConfs. Having HadoopRDD accept this optional closure eliminates the need for the HadoopFileRDD added earlier. It makes the HadoopRDD more general, in that the caller can specify any JobConf initialization flow.
| | * | Remove the new HadoopRDD constructor from SparkContext API, plus some minor ↵Harvey Feng2013-10-122-27/+3
| | | | | | | | | | | | | | | | style changes.
| | * | Add an optional closure parameter to HadoopRDD instantiation to used when ↵Harvey Feng2013-10-102-53/+48
| | | | | | | | | | | | | | | | creating any local JobConfs.
| * | | Merge pull request #53 from witgo/masterMatei Zaharia2013-10-111-0/+4
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | Add a zookeeper compile dependency to fix build in maven Add a zookeeper compile dependency to fix build in maven
| | * | | Add a zookeeper compile dependency to fix build in mavenLiGuoqiang2013-10-111-0/+4
| | | | |
| * | | | Merge pull request #32 from mridulm/masterMatei Zaharia2013-10-1112-29/+93
| |\ \ \ \ | | |/ / / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | Address review comments, move to incubator spark Also includes a small fix to speculative execution. <edit> Continued from https://github.com/mesos/spark/pull/914 </edit>
| | * | | - Allow for finer control of cleanerMridul Muralidharan2013-10-0612-29/+93
| | | |/ | | |/| | | | | | | | | | | | | - Address review comments, move to incubator spark - Also includes a change to speculation - including preventing exceptions in rare cases.
* | | | Fixed PairRDDFunctionsSuite after removing InterruptibleRDD.Reynold Xin2013-10-121-1/+1
| | | |
* | | | Job cancellation: address Matei's code review feedback.Reynold Xin2013-10-1217-216/+248
| | | |
* | | | Job cancellation: addressed code review feedback round 2 from Kay.Reynold Xin2013-10-113-44/+47
| | | |
* | | | Fixed dagscheduler suite because of a logging message change.Reynold Xin2013-10-111-1/+1
| | | |
* | | | Job cancellation: addressed code review feedback from Kay.Reynold Xin2013-10-1112-80/+86
| | | |
* | | | Making takeAsync and collectAsync deterministic.Reynold Xin2013-10-113-19/+15
| | | |
* | | | Properly handle interrupted exception in FutureAction.Reynold Xin2013-10-111-7/+5
| | | |
* | | | Merge branch 'master' of github.com:apache/incubator-spark into killReynold Xin2013-10-1035-212/+2106
|\| | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala
| * | | Merge remote-tracking branch 'tgravescs/sparkYarnDistCache'Matei Zaharia2013-10-101-3/+14
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Closes #11 Conflicts: docs/running-on-yarn.md yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
| | * | | Adding in the --addJars option to make SparkContext.addJar work on yarn and ↵tgravescs2013-10-031-3/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | cleanup the classpaths
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-11 06:15:42 +0000