aboutsummaryrefslogtreecommitdiff
path: root/project
Commit message (Collapse)AuthorAgeFilesLines
...
| * | | | | | | | | | | | Fixing spark streaming example and a bug in examples build.Patrick Wendell2013-10-151-0/+1
| | |/ / / / / / / / / / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Examples assembly included a log4j.properties which clobbered Spark's - Example had an error where some classes weren't serializable - Did some other clean-up in this example
| * | | | | | | | | | | Merge pull request #19 from aarondav/master-zkMatei Zaharia2013-10-101-0/+1
| |\ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Standalone Scheduler fault tolerance using ZooKeeper This patch implements full distributed fault tolerance for standalone scheduler Masters. There is only one master Leader at a time, which is actively serving scheduling requests. If this Leader crashes, another master will eventually be elected, reconstruct the state from the first Master, and continue serving scheduling requests. Leader election is performed using the ZooKeeper leader election pattern. We try to minimize the use of ZooKeeper and the assumptions about ZooKeeper's behavior, so there is a layer of retries and session monitoring on top of the ZooKeeper client. Master failover follows directly from the single-node Master recovery via the file system (patch d5a96fe), save that the Master state is stored in ZooKeeper instead. Configuration: By default, no recovery mechanism is enabled (spark.deploy.recoveryMode = NONE). By setting spark.deploy.recoveryMode to ZOOKEEPER and setting spark.deploy.zookeeper.url to an appropriate ZooKeeper URL, ZooKeeper recovery mode is enabled. By setting spark.deploy.recoveryMode to FILESYSTEM and setting spark.deploy.recoveryDirectory to an appropriate directory accessible by the Master, we will keep the behavior of from d5a96fe. Additionally, places where a Master could be specificied by a spark:// url can now take comma-delimited lists to specify backup masters. Note that this is only used for registration of NEW Workers and application Clients. Once a Worker or Client has registered with the Master Leader, it is "in the system" and will never need to register again.
| | * | | | | | | | | | | Standalone Scheduler fault tolerance using ZooKeeperAaron Davidson2013-09-261-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch implements full distributed fault tolerance for standalone scheduler Masters. There is only one master Leader at a time, which is actively serving scheduling requests. If this Leader crashes, another master will eventually be elected, reconstruct the state from the first Master, and continue serving scheduling requests. Leader election is performed using the ZooKeeper leader election pattern. We try to minimize the use of ZooKeeper and the assumptions about ZooKeeper's behavior, so there is a layer of retries and session monitoring on top of the ZooKeeper client. Master failover follows directly from the single-node Master recovery via the file system (patch 194ba4b8), save that the Master state is stored in ZooKeeper instead. Configuration: By default, no recovery mechanism is enabled (spark.deploy.recoveryMode = NONE). By setting spark.deploy.recoveryMode to ZOOKEEPER and setting spark.deploy.zookeeper.url to an appropriate ZooKeeper URL, ZooKeeper recovery mode is enabled. By setting spark.deploy.recoveryMode to FILESYSTEM and setting spark.deploy.recoveryDirectory to an appropriate directory accessible by the Master, we will keep the behavior of from 194ba4b8. Additionally, places where a Master could be specificied by a spark:// url can now take comma-delimited lists to specify backup masters. Note that this is only used for registration of NEW Workers and application Clients. Once a Worker or Client has registered with the Master Leader, it is "in the system" and will never need to register again. Forthcoming: Documentation, tests (! - only ad hoc testing has been performed so far) I do not intend for this commit to be merged until tests are added, but this patch should still be mostly reviewable until then.
* | | | | | | | | | | | | Updating to latest akka 2.2.3, which fixes our only failing Driver SuitePrashant Sharma2013-10-241-4/+4
| | | | | | | | | | | | |
* | | | | | | | | | | | | Merge branch 'scala-2.10' of github.com:ScrapCodes/spark into scala-2.10Prashant Sharma2013-10-101-8/+14
|\ \ \ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala project/SparkBuild.scala
| * | | | | | | | | | | | | Merge branch 'master' into wip-merge-masterPrashant Sharma2013-10-081-6/+8
| |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: bagel/pom.xml core/pom.xml core/src/test/scala/org/apache/spark/ui/UISuite.scala examples/pom.xml mllib/pom.xml pom.xml project/SparkBuild.scala repl/pom.xml streaming/pom.xml tools/pom.xml In scala 2.10, a shorter representation is used for naming artifacts so changed to shorter scala version for artifacts and made it a property in pom.
| | * | | | | | | | | | | | Merge pull request #31 from sundeepn/branch-0.8Reynold Xin2013-10-071-5/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Resolving package conflicts with hadoop 0.23.9 Hadoop 0.23.9 is having a package conflict with easymock's dependencies. (cherry picked from commit 023e3fdf008b3194a36985a07923df9aaf64e520) Signed-off-by: Reynold Xin <rxin@apache.org>
| * | | | | | | | | | | | | Merge branch 'master' into scala-2.10Prashant Sharma2013-10-051-0/+3
| |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/DistributedSuite.scala project/SparkBuild.scala
| | * | | | | | | | | | | | ask ivy/sbt to check local maven repo under ~/.m2Du Li2013-10-011-0/+3
| | | | | | | | | | | | | |
| * | | | | | | | | | | | | Merge branch 'master' into scala-2.10Prashant Sharma2013-10-011-2/+3
| |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala docs/_config.yml project/SparkBuild.scala repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
| | * | | | | | | | | | | | Removed scala -optimize flag.Reynold Xin2013-09-261-1/+1
| | |/ / / / / / / / / / /
| | * | | | | | | | | | | Merge pull request #930 from holdenk/masterReynold Xin2013-09-261-1/+1
| | |\ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add mapPartitionsWithIndex
| | | * | | | | | | | | | | Fix build on ubuntuHolden Karau2013-09-141-1/+1
| | | | | | | | | | | | | |
| | * | | | | | | | | | | | Update build version in masterPatrick Wendell2013-09-241-1/+1
| | | |_|_|_|_|_|_|_|_|_|/ | | |/| | | | | | | | | |
* | / | | | | | | | | | | scala 2.10 requires Java 1.6,Martin Weindel2013-10-051-3/+3
|/ / / / / / / / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | using Scala 2.10.3, resolved maven-scala-plugin warning
* | | | | | | | | | | | Sync with master and some build fixesPrashant Sharma2013-09-261-8/+8
|\| | | | | | | | | | |
| * | | | | | | | | | | Bumping Mesos version to 0.13.0Patrick Wendell2013-09-151-1/+1
| |/ / / / / / / / / /
* | | | | | | | | | | fixed maven build for scala 2.10Prashant Sharma2013-09-261-2/+1
| | | | | | | | | | |
* | | | | | | | | | | Akka 2.2 migrationPrashant Sharma2013-09-221-9/+9
| | | | | | | | | | |
* | | | | | | | | | | Merge branch 'master' of git://github.com/mesos/spark into scala-2.10Prashant Sharma2013-09-152-6/+31
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/SparkContext.scala project/SparkBuild.scala
| * | | | | | | | | | Merge pull request #919 from mateiz/jets3tPatrick Wendell2013-09-111-0/+1
| |\ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Add explicit jets3t dependency, which is excluded in hadoop-client
| | * | | | | | | | | | Add explicit jets3t dependency, which is excluded in hadoop-clientMatei Zaharia2013-09-101-0/+1
| | | | | | | | | | | |
| * | | | | | | | | | | Fix HDFS access bug with assembly build.Patrick Wendell2013-09-101-0/+1
| |/ / / / / / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Due to this change in HDFS: https://issues.apache.org/jira/browse/HADOOP-7549 there is a bug when using the new assembly builds. The symptom is that any HDFS access results in an exception saying "No filesystem for scheme 'hdfs'". This adds a merge strategy in the assembly build which fixes the problem.
| * | | | | | | | | | Merge pull request #906 from pwendell/ganglia-sinkPatrick Wendell2013-09-081-0/+1
| |\ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Clean-up of Metrics Code/Docs and Add Ganglia Sink
| | * | | | | | | | | | Ganglia sinkPatrick Wendell2013-09-081-0/+1
| | | | | | | | | | | |
| * | | | | | | | | | | Merge pull request #908 from pwendell/masterMatei Zaharia2013-09-081-1/+7
| |\ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | Fix target JVM version in scala build
| | * | | | | | | | | | | Fix target JVM version in scala buildPatrick Wendell2013-09-081-1/+7
| | |/ / / / / / / / / /
| * | | | | | | | | | | Merge pull request #904 from pwendell/masterPatrick Wendell2013-09-071-1/+18
| |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Adding Apache license to two files
| | * | | | | | | | | | Adding Apache license to two filesPatrick Wendell2013-09-071-1/+18
| | |/ / / / / / / / /
| * / / / / / / / / / Minor YARN build cleanupsJey Kottalam2013-09-061-2/+2
| |/ / / / / / / / /
* | | | | | | | | | Fixed repl suitePrashant Sharma2013-09-151-5/+5
| | | | | | | | | |
* | | | | | | | | | Merged with masterPrashant Sharma2013-09-063-69/+142
|\| | | | | | | | |
| * | | | | | | | | Add Apache parent POMMatei Zaharia2013-09-021-0/+5
| | | | | | | | | |
| * | | | | | | | | Fix some URLsMatei Zaharia2013-09-011-2/+2
| | | | | | | | | |
| * | | | | | | | | Initial work to rename package to org.apache.sparkMatei Zaharia2013-09-011-4/+8
| | | | | | | | | |
| * | | | | | | | | Update Maven build to create assemblies expected by new scriptsMatei Zaharia2013-08-291-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This includes the following changes: - The "assembly" package now builds in Maven by default, and creates an assembly containing both hadoop-client and Spark, unlike the old BigTop distribution assembly that skipped hadoop-client - There is now a bigtop-dist package to build the old BigTop assembly - The repl-bin package is no longer built by default since the scripts don't reply on it; instead it can be enabled with -Prepl-bin - Py4J is now included in the assembly/lib folder as a local Maven repo, so that the Maven package can link to it - run-example now adds the original Spark classpath as well because the Maven examples assembly lists spark-core and such as provided - The various Maven projects add a spark-yarn dependency correctly
| * | | | | | | | | Provide more memory for testsMatei Zaharia2013-08-291-1/+1
| | | | | | | | | |
| * | | | | | | | | Change build and run instructions to use assembliesMatei Zaharia2013-08-293-22/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
| * | | | | | | | | Revert "Merge pull request #841 from rxin/json"Reynold Xin2013-08-261-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit 1fb1b0992838c8cdd57eec45793e67a0490f1a52, reversing changes made to c69c48947d5102c81a9425cb380d861c3903685c.
| * | | | | | | | | Upgrade SBT IDE project generatorsJey Kottalam2013-08-231-2/+2
| | | | | | | | | |
| * | | | | | | | | Fix SBT generation of IDE project filesJey Kottalam2013-08-231-5/+12
| | | | | | | | | |
| * | | | | | | | | Re-add removed dependency on 'commons-daemon'Jey Kottalam2013-08-221-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes SBT build under Hadoop 0.23.9 and 2.0.4
| * | | | | | | | | Merge pull request #855 from jey/update-build-docsMatei Zaharia2013-08-221-4/+3
| |\ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | Update build docs
| | * | | | | | | | | Remove references to unsupported Hadoop versionsJey Kottalam2013-08-211-4/+3
| | | | | | | | | | |
| * | | | | | | | | | Merge pull request #854 from markhamstra/pomUpdateMatei Zaharia2013-08-221-4/+1
| |\ \ \ \ \ \ \ \ \ \ | | |/ / / / / / / / / | |/| | | | | | | | | Synced sbt and maven builds to use the same dependencies, etc.
| | * | | | | | | | | Synced sbt and maven buildsMark Hamstra2013-08-211-4/+1
| | | | | | | | | | |
| * | | | | | | | | | Downgraded default build hadoop version to 1.0.4.Reynold Xin2013-08-211-1/+1
| |/ / / / / / / / /
| * | | | | | | | | Merge remote-tracking branch 'jey/hadoop-agnostic'Matei Zaharia2013-08-201-41/+35
| |\ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/PairRDDFunctions.scala
| | * | | | | | | | | Update SBT build to use simpler fix for Hadoop 0.23.9Jey Kottalam2013-08-191-11/+2
| | | | | | | | | | |
| | * | | | | | | | | Rename YARN build flag to SPARK_WITH_YARNJey Kottalam2013-08-161-5/+7
| | | | | | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-16 12:56:23 +0000