aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Add docs for standalone scheduler fault toleranceAaron Davidson2013-10-083-4/+48
| | | | Also fix a couple HTML/Markdown issues in other files.
* Revert change to spark-classAaron Davidson2013-10-082-3/+8
| | | | Also adds comment about how to configure for FaultToleranceTest.
* Add license agreements to dockerfilesAaron Davidson2013-10-087-1/+120
|
* Change url format to spark://host1:port1,host2:port2Aaron Davidson2013-10-065-7/+14
| | | | | This replaces the format of spark://host1:port1,spark://host2:port2 and is more consistent with ZooKeeper's zk:// urls.
* Add end-to-end test for standalone scheduler fault toleranceAaron Davidson2013-10-0512-2/+480
| | | | Docker files drawn mostly from Matt Masse. Some updates from Andre Schumacher.
* Address Matei's commentsAaron Davidson2013-10-0511-42/+41
|
* Fix race conditions during recoveryAaron Davidson2013-10-048-52/+122
| | | | | | | | | | One major change was the use of messages instead of raw functions as the parameter of Akka scheduled timers. Since messages are serialized, unlike raw functions, the behavior is easier to think about and doesn't cause race conditions when exceptions are thrown. Another change is to avoid using global pointers that might change without a lock.
* Add license noticesAaron Davidson2013-09-266-3/+86
|
* Standalone Scheduler fault tolerance using ZooKeeperAaron Davidson2013-09-2625-170/+720
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch implements full distributed fault tolerance for standalone scheduler Masters. There is only one master Leader at a time, which is actively serving scheduling requests. If this Leader crashes, another master will eventually be elected, reconstruct the state from the first Master, and continue serving scheduling requests. Leader election is performed using the ZooKeeper leader election pattern. We try to minimize the use of ZooKeeper and the assumptions about ZooKeeper's behavior, so there is a layer of retries and session monitoring on top of the ZooKeeper client. Master failover follows directly from the single-node Master recovery via the file system (patch 194ba4b8), save that the Master state is stored in ZooKeeper instead. Configuration: By default, no recovery mechanism is enabled (spark.deploy.recoveryMode = NONE). By setting spark.deploy.recoveryMode to ZOOKEEPER and setting spark.deploy.zookeeper.url to an appropriate ZooKeeper URL, ZooKeeper recovery mode is enabled. By setting spark.deploy.recoveryMode to FILESYSTEM and setting spark.deploy.recoveryDirectory to an appropriate directory accessible by the Master, we will keep the behavior of from 194ba4b8. Additionally, places where a Master could be specificied by a spark:// url can now take comma-delimited lists to specify backup masters. Note that this is only used for registration of NEW Workers and application Clients. Once a Worker or Client has registered with the Master Leader, it is "in the system" and will never need to register again. Forthcoming: Documentation, tests (! - only ad hoc testing has been performed so far) I do not intend for this commit to be merged until tests are added, but this patch should still be mostly reviewable until then.
* Standalone Scheduler fault recoveryAaron Davidson2013-09-2616-76/+459
| | | | | | | | | | | | | | | | | | | Implements a basic form of Standalone Scheduler fault recovery. In particular, this allows faults to be manually recovered from by means of restarting the Master process on the same machine. This is the majority of the code necessary for general fault tolerance, which will first elect a leader and then recover the Master state. In order to enable fault recovery, the Master will persist a small amount of state related to the registration of Workers and Applications to disk. If the Master is started and sees that this state is still around, it will enter Recovery mode, during which time it will not schedule any new Executors on Workers (but it does accept the registration of new Clients and Workers). At this point, the Master attempts to reconnect to all Workers and Client applications that were registered at the time of failure. After confirming either the existence or nonexistence of all such nodes (within a certain timeout), the Master will exit Recovery mode and resume normal scheduling.
* Merge pull request #16 from pwendell/masterReynold Xin2013-09-261-1/+1
|\ | | | | | | Bug fix in master build
| * Bug fix in master buildPatrick Wendell2013-09-261-1/+1
| |
* | Merge pull request #14 from kayousterhout/untangle_schedulerReynold Xin2013-09-2634-71/+62
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Improved organization of scheduling packages. This commit does not change any code -- only file organization. Please let me know if there was some masterminded strategy behind the existing organization that I failed to understand! There are two components of this change: (1) Moving files out of the cluster package, and down a level to the scheduling package. These files are all used by the local scheduler in addition to the cluster scheduler(s), so should not be in the cluster package. As a result of this change, none of the files in the local package reference files in the cluster package. (2) Moving the mesos package to within the cluster package. The mesos scheduling code is for a cluster, and represents a specific case of cluster scheduling (the Mesos-related classes often subclass cluster scheduling classes). Thus, the most logical place for it seems to be within the cluster package. The one thing about the scheduling code that seems a little funny to me is the naming of the SchedulerBackends. The StandaloneSchedulerBackend is not just for Standalone mode, but instead is used by Mesos coarse grained mode and Yarn, and the backend that *is* just for Standalone mode is instead called SparkDeploySchedulerBackend. I didn't change this because I wasn't sure if there was a reason for this naming that I'm just not aware of.
| * | Improved organization of scheduling packages.Kay Ousterhout2013-09-2534-71/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit does not change any code -- only file organization. There are two components of this change: (1) Moving files out of the cluster package, and down a level to the scheduling package. These files are all used by the local scheduler in addition to the cluster scheduler(s), so should not be in the cluster package. As a result of this change, none of the files in the local package reference files in the cluster package. (2) Moving the mesos package to within the cluster package. The mesos scheduling code is for a cluster, and represents a specific case of cluster scheduling (the Mesos-related classes often subclass cluster scheduling classes). Thus, the most logical place for it is within the cluster package.
* | | Merge pull request #670 from jey/ec2-ssh-improvementsReynold Xin2013-09-261-26/+80
|\ \ \ | | | | | | | | EC2 SSH improvements
| * | | Clarify error messages on SSH failureJey Kottalam2013-09-111-6/+21
| | | |
| * | | Generate new SSH key for the cluster, make "--identity-file" optionalJey Kottalam2013-09-061-21/+37
| | | |
| * | | Construct shell commands as sequences for safety and composabilityJey Kottalam2013-09-061-11/+34
| | | |
* | | | Merge pull request #930 from holdenk/masterReynold Xin2013-09-262-1/+11
|\ \ \ \ | | | | | | | | | | Add mapPartitionsWithIndex
| * | | | Fix formatting :)Holden Karau2013-09-231-4/+5
| | | | |
| * | | | Switch indent from 2 to 4 spacesHolden Karau2013-09-221-2/+2
| | | | |
| * | | | Fix build on ubuntuHolden Karau2013-09-141-1/+1
| | | | |
| * | | | Merge branch 'master' of https://github.com/mesos/sparkHolden Karau2013-09-142-5/+12
| |\ \ \ \
| * | | | | Make mapPartitionsWithIndex work with JavaRDD'sHolden Karau2013-09-141-2/+3
| | | | | |
| * | | | | Start of working on SPARK-615Holden Karau2013-09-111-0/+8
| | | | | |
* | | | | | Merge pull request #7 from wannabeast/memorystore-fixesReynold Xin2013-09-261-6/+8
|\ \ \ \ \ \ | |_|_|_|_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | some minor fixes to MemoryStore This is a repeat of #5, moved to its own branch in my repo. This makes all updates to on ; it skips on synchronizing the reads where it can get away with it.
| * | | | | Synchronize on "entries" the remaining update to "currentMemory".Mike2013-09-191-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | Make "currentMemory" @volatile, so that it's reads in ensureFreeSpace() are atomic and up-to-date--i.e., currentMemory can't increase while putLock is held (though it could decrease, which would only help ensureFreeSpace()).
| * | | | | Set currentMemory to 0 in clear().Mike2013-09-111-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | Remove unnecessary entries.get() call.
| * | | | | Remove MemoryStore$Entry.dropPending, unused as of 42e0a68082.Mike2013-09-101-1/+1
| | | | | |
* | | | | | Merge pull request #9 from rxin/limitPatrick Wendell2013-09-262-10/+66
|\ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | Smarter take/limit implementation.
| * | | | | | Smarter take/limit implementation.Reynold Xin2013-09-202-10/+66
| | | | | | |
* | | | | | | Merge remote-tracking branch 'apache-github/pr/13' into HEADPatrick Wendell2013-09-2414-15/+15
|\ \ \ \ \ \ \ | |_|_|_|_|_|/ |/| | | | | |
| * | | | | | Update build version in masterPatrick Wendell2013-09-2414-15/+15
|/ / / / / /
* | | | | | Merge remote-tracking branch 'pr/12'Reynold Xin2013-09-232-4/+6
|\ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | Fix spacing so java.io.tmpdir doesn't run on with SPARK_JAVA_OPTS
| * | | | | | Fix spacing so that the java.io.tmpdir doesn't run on with SPARK_JAVA_OPTSY.CORP.YAHOO.COM\tgraves2013-09-232-4/+6
| |/ / / / /
* | | | | | Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/incubator-sparkReynold Xin2013-09-230-0/+0
|\| | | | |
| * | | | | Merge branch 'master' of github.com:mesos/sparkReynold Xin2013-09-2096-543/+2068
| |\ \ \ \ \ | | |/ / / / | |/| | | |
* | | | | | Merge branch 'master' of github.com:markhamstra/incubator-sparkReynold Xin2013-09-231-1/+0
|\ \ \ \ \ \
| * | | | | | Removed repetative import; fixes hidden definition compiler warning.Mark Hamstra2013-09-031-1/+0
| |/ / / / /
* | | | | | Merge branch 'master' of github.com:mesos/sparkReynold Xin2013-09-237-64/+123
|\ \ \ \ \ \
| * \ \ \ \ \ Merge pull request #928 from jerryshao/fairscheduler-refactorReynold Xin2013-09-221-43/+56
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | Refactor FairSchedulableBuilder
| | * | | | | | Change Exception to NoSuchElementException and minor style fixjerryshao2013-09-221-6/+7
| | | | | | | |
| | * | | | | | Remove infix style and othersjerryshao2013-09-221-10/+8
| | | | | | | |
| | * | | | | | Refactor FairSchedulableBuilder:jerryshao2013-09-221-39/+53
| |/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Configuration can be read from classpath if not set explicitly. 2. Add missing close handler.
| * | | | | | Merge pull request #937 from jerryshao/localProperties-fixReynold Xin2013-09-212-2/+50
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | Fix PR926 local properties issues in Spark Streaming like scenarios
| | * | | | | | Add barrier for local properties unit test and fix some stylesjerryshao2013-09-222-3/+11
| | | | | | | |
| | * | | | | | Fix issue when local properties pass from parent to child threadjerryshao2013-09-182-2/+42
| | | | | | | |
| * | | | | | | Merge pull request #941 from ilikerps/masterReynold Xin2013-09-211-6/+6
| |\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | Add "org.apache." prefix to packages in spark-class
| | * | | | | | | Add "org.apache." prefix to packages in spark-classAaron Davidson2013-09-201-6/+6
| | | |_|/ / / / | | |/| | | | | | | | | | | | | | | | | | | | | Lacking this, the if/case statements never trigger on Spark 0.8.0+.
| * | | | | | | Merge pull request #940 from ankurdave/clear-port-properties-after-testsReynold Xin2013-09-212-9/+7
| |\ \ \ \ \ \ \ | | |/ / / / / / | |/| | | | | | After unit tests, clear port properties unconditionally
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-19 05:03:58 +0000