aboutsummaryrefslogtreecommitdiff
path: root/yarn
Commit message (Collapse)AuthorAgeFilesLines
* Merge remote-tracking branch 'origin/master' into conf2Matei Zaharia2013-12-295-11/+19
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala core/src/main/scala/org/apache/spark/scheduler/local/LocalScheduler.scala core/src/main/scala/org/apache/spark/util/MetadataCleaner.scala core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala new-yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala streaming/src/test/scala/org/apache/spark/streaming/WindowOperationsSuite.scala
| * Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarnliguoqiang2013-12-261-2/+1
| |
| * Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarnliguoqiang2013-12-264-8/+11
| |
| * Merge remote branch 'upstream/master' into consolidate_schedulersKay Ousterhout2013-12-204-17/+15
| |\ | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
| * \ Merge master into 127Aaron Davidson2013-12-0811-365/+936
| |\ \
| * | | Fixed naming issues and added back ability to specify max task failures.Kay Ousterhout2013-11-131-3/+3
| | | |
| * | | Merge remote-tracking branch 'upstream/master' into consolidate_schedulersKay Ousterhout2013-11-136-172/+646
| |\ \ \ | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/scheduler/ClusterScheduler.scala
| * | | | Deduplicate Local and Cluster schedulers.Kay Ousterhout2013-10-301-3/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The code in LocalScheduler/LocalTaskSetManager was nearly identical to the code in ClusterScheduler/ClusterTaskSetManager. The redundancy made making updating the schedulers unnecessarily painful and error- prone. This commit combines the two into a single TaskScheduler/ TaskSetManager.
* | | | | Various fixes to configuration codeMatei Zaharia2013-12-283-54/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Got rid of global SparkContext.globalConf - Pass SparkConf to serializers and compression codecs - Made SparkConf public instead of private[spark] - Improved API of SparkContext and SparkConf - Switched executor environment vars to be passed through SparkConf - Fixed some places that were still using system properties - Fixed some tests, though others are still failing This still fails several tests in core, repl and streaming, likely due to properties not being set or cleared correctly (some of the tests run fine in isolation).
* | | | | spark-544, introducing SparkConf and related configuration overhaul.Prashant Sharma2013-12-256-17/+17
| |_|_|/ |/| | |
* | | | Merge pull request #265 from markhamstra/scala.binary.versionPatrick Wendell2013-12-151-4/+4
|\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | DRY out the POMs with scala.binary.version ...instead of hard-coding 2.10 repeatedly. As long as it's not a `<project>`-level `<artifactId>`, I think that we are okay parameterizing these.
| * | | | Use scala.binary.version in POMsMark Hamstra2013-12-151-4/+4
| | | | |
* | | | | Merge pull request #257 from tgravescs/sparkYarnFixNameReynold Xin2013-12-151-0/+1
|\ \ \ \ \ | |/ / / / |/| | | | | | | | | | | | | | | | | | | Fix the --name option for Spark on Yarn Looks like the --name option accidentally got broken in one of the merges. The Client hangs if the --name option is used right now.
| * | | | Fix the --name option for Spark on YarnThomas Graves2013-12-121-0/+1
| | |_|/ | |/| |
* | | | Merge branch 'master' into akka-bug-fixPrashant Sharma2013-12-114-309/+445
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/pom.xml core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala pom.xml project/SparkBuild.scala streaming/pom.xml yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
| * | | Merge remote-tracking branch 'origin/master' into yarn-2.2Harvey Feng2013-11-265-19/+433
| |\ \ \ | | | | | | | | | | | | | | | | | | | | Conflicts: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
| * | | | A few more style fixes in `yarn` package.Harvey Feng2013-11-233-45/+71
| | | | |
| * | | | Merge branch 'master' into yarn-cleanupHarvey Feng2013-11-216-51/+81
| |\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala yarn/src/main/scala/org/apache/spark/deploy/yarn/WorkerRunnable.scala yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
| * | | | | Misc style changes in the 'yarn' package.Harvey Feng2013-11-174-273/+376
| | | | | |
* | | | | | Style fixes and addressed review comments at #221Prashant Sharma2013-12-101-4/+4
| | | | | |
* | | | | | fixed yarn buildPrashant Sharma2013-12-092-11/+8
| | | | | |
* | | | | | Incorporated Patrick's feedback comment on #211 and made maven ↵Prashant Sharma2013-12-071-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | build/dep-resolution atleast a bit faster.
* | | | | | A left over akka -> akka.tcp changesPrashant Sharma2013-12-061-1/+1
| | | | | |
* | | | | | Merge branch 'master' into wip-scala-2.10Prashant Sharma2013-11-275-21/+434
|\ \ \ \ \ \ | | |_|/ / / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala core/src/main/scala/org/apache/spark/rdd/MapPartitionsWithContextRDD.scala core/src/main/scala/org/apache/spark/rdd/RDD.scala python/pyspark/rdd.py
| * | | | | Add YarnClientClusterScheduler and Backend.Raymond Liu2013-11-225-21/+434
| | |/ / / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | With this scheduler, the user application is launched locally, While the executor will be launched by YARN on remote nodes. This enables spark-shell to run upon YARN.
* | | | | Merge branch 'master' of github.com:apache/incubator-spark into scala-2.10-tempPrashant Sharma2013-11-217-68/+90
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/util/collection/PrimitiveVector.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
| * | | | Merge branch 'master' into removesemicolonscalaHenry Saputra2013-11-193-29/+58
| |\ \ \ \
| | * | | | Impove Spark on Yarn Error handlingtgravescs2013-11-193-29/+58
| | |/ / /
| * | | | Another set of changes to remove unnecessary semicolon (;) from Scala code.Henry Saputra2013-11-194-5/+5
| | | | | | | | | | | | | | | | | | | | Passed the sbt/sbt compile and test
| * | | | Remove the semicolons at the end of Scala code to make it more pure Scala code.Henry Saputra2013-11-195-26/+21
| |/ / / | | | | | | | | | | | | | | | | | | | | | | | | Also remove unused imports as I found them along the way. Remove return statements when returning value in the Scala code. Passing compile and tests.
| * | / Simple cleanup on Spark's Scala code while testing core and yarn modules:Henry Saputra2013-11-154-8/+6
| | |/ | |/| | | | | | | | | | | | | -) Remove some of unused imports as I found them -) Remove ";" in the imports statements -) Remove () at the end of method call like size that does not have size effect.
* | | Merge branch 'master' into scala-2.10Raymond Liu2013-11-146-172/+646
|\| |
| * | Allow spark on yarn to be run from HDFS. Allows the spark.jar, app.jar, and ↵tgravescs2013-11-046-172/+646
| |/ | | | | | | log4j.properties to be put into hdfs.
* | Merge branch 'master' into scala-2.10Raymond Liu2013-11-135-51/+278
|\|
| * Fix the Worker to use CoarseGrainedExecutorBackend and modify classpath to ↵tgravescs2013-10-212-10/+29
| | | | | | | | | | | | be explicit about inclusion of spark.jar and app.jar
| * Fix yarn buildtgravescs2013-10-161-2/+2
| |
| * Merge remote-tracking branch 'tgravescs/sparkYarnDistCache'Matei Zaharia2013-10-104-45/+253
| |\ | | | | | | | | | | | | | | | | | | | | | Closes #11 Conflicts: docs/running-on-yarn.md yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
| | * Adding in the --addJars option to make SparkContext.addJar work on yarn and ↵tgravescs2013-10-033-20/+34
| | | | | | | | | | | | | | | | | | cleanup the classpaths
| | * Support distributed cache files and archives on spark on yarn and attempt to ↵Y.CORP.YAHOO.COM\tgraves2013-09-234-29/+226
| | | | | | | | | | | | cleanup the staging directory on exit
* | | Merge branch 'master' into wip-merge-masterPrashant Sharma2013-10-081-3/+3
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: bagel/pom.xml core/pom.xml core/src/test/scala/org/apache/spark/ui/UISuite.scala examples/pom.xml mllib/pom.xml pom.xml project/SparkBuild.scala repl/pom.xml streaming/pom.xml tools/pom.xml In scala 2.10, a shorter representation is used for naming artifacts so changed to shorter scala version for artifacts and made it a property in pom.
| * | Merging build changes in from 0.8Patrick Wendell2013-10-051-3/+3
| | |
* | | Merge branch 'master' into scala-2.10Prashant Sharma2013-10-052-1/+7
|\| | | | | | | | | | | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/DistributedSuite.scala project/SparkBuild.scala
| * | Add default value to usage statementtgravescs2013-10-031-1/+1
| | |
| * | Allow users to set the application name for Spark on Yarntgravescs2013-10-022-1/+7
| | |
* | | Merge branch 'master' into scala-2.10Prashant Sharma2013-10-013-5/+7
|\| | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala docs/_config.yml project/SparkBuild.scala repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
| * | Update build version in masterPatrick Wendell2013-09-241-1/+1
| | |
| * | Fix spacing so that the java.io.tmpdir doesn't run on with SPARK_JAVA_OPTSY.CORP.YAHOO.COM\tgraves2013-09-232-4/+6
| |/
* | fixed maven build for scala 2.10Prashant Sharma2013-09-261-2/+2
| |
* | Akka 2.2 migrationPrashant Sharma2013-09-221-1/+1
|/
* Use different Hadoop version for YARN artifacts.Patrick Wendell2013-09-131-0/+5
| | | | | This uses a seperate Hadoop version for YARN artifact. This means when people link against spark-yarn, things will resolve correctly.
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-07 14:29:47 +0000