aboutsummaryrefslogtreecommitdiff
path: root/core
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #374 from mateiz/completenessReynold Xin2014-01-096-6/+90
|\ | | | | | | | | | | Add some missing Java API methods These are primarily for setting job groups, canceling jobs, and setting names on RDDs. Seemed like useful stuff to expose in Java.
| * Add some missing Java API methodsMatei Zaharia2014-01-096-6/+90
| |
* | Merge pull request #294 from RongGu/masterReynold Xin2014-01-091-1/+6
|\ \ | | | | | | | | | | | | | | | | | | | | | Bug fixes for updating the RDD block's memory and disk usage information Bug fixes for updating the RDD block's memory and disk usage information. From the code context, we can find that the memSize and diskSize here are both always equal to the size of the block. Actually, they never be zero. Thus, the logic here is wrong for recording the block usage in BlockStatus, especially for the blocks which are dropped from memory to ensure space for the new input rdd blocks. I have tested it that this would cause the storage metrics shown in the Storage webpage wrong and misleading. With this patch, the metrics will be okay. Finally, Merry Christmas, guys:)
| * \ Merge remote branch 'upstream/master'walker2014-01-0925-104/+119
| |\ \
| * | | add inline commentswalker2014-01-071-1/+1
| | | |
| * | | add inline commentswalker2014-01-071-0/+4
| | | |
| * | | Merge remote branch 'upstream/master'walker2014-01-07140-2742/+3418
| |\ \ \
| * | | | Bug fixes for updating the RDD block's memory and disk usage informationwalker2013-12-251-1/+2
| | | | |
* | | | | Merge pull request #293 from pwendell/standalone-driverPatrick Wendell2014-01-0930-148/+1471
|\ \ \ \ \ | |_|_|_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SPARK-998: Support Launching Driver Inside of Standalone Mode [NOTE: I need to bring the tests up to date with new changes, so for now they will fail] This patch provides support for launching driver programs inside of a standalone cluster manager. It also supports monitoring and re-launching of driver programs which is useful for long running, recoverable applications such as Spark Streaming jobs. For those jobs, this patch allows a deployment mode which is resilient to the failure of any worker node, failure of a master node (provided a multi-master setup), and even failures of the applicaiton itself, provided they are recoverable on a restart. Driver information, such as the status and logs from a driver, is displayed in the UI There are a few small TODO's here, but the code is generally feature-complete. They are: - Bring tests up to date and add test coverage - Restarting on failure should be optional and maybe off by default. - See if we can re-use akka connections to facilitate clients behind a firewall A sensible place to start for review would be to look at the `DriverClient` class which presents users the ability to launch their driver program. I've also added an example program (`DriverSubmissionTest`) that allows you to test this locally and play around with killing workers, etc. Most of the code is devoted to persisting driver state in the cluster manger, exposing it in the UI, and dealing correctly with various types of failures. Instructions to test locally: - `sbt/sbt assembly/assembly examples/assembly` - start a local version of the standalone cluster manager ``` ./spark-class org.apache.spark.deploy.client.DriverClient \ -j -Dspark.test.property=something \ -e SPARK_TEST_KEY=SOMEVALUE \ launch spark://10.99.1.14:7077 \ ../path-to-examples-assembly-jar \ org.apache.spark.examples.DriverSubmissionTest 1000 some extra options --some-option-here -X 13 ``` - Go in the UI and make sure it started correctly, look at the output etc - Kill workers, the driver program, masters, etc.
| * | | | Some usability improvementsPatrick Wendell2014-01-093-25/+62
| | | | |
| * | | | Adding polling to driver submission client.Patrick Wendell2014-01-086-68/+132
| | | | |
| * | | | Adding mockito to maven buildPatrick Wendell2014-01-081-0/+5
| | | | |
| * | | | Merge remote-tracking branch 'apache-github/master' into standalone-driverPatrick Wendell2014-01-0825-103/+118
| |\ \ \ \ | | | |_|/ | | |/| | | | | | | | | | | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/deploy/JsonProtocolSuite.scala pom.xml
| * | | | Show more helpful information in UIPatrick Wendell2014-01-084-9/+18
| | | | |
| * | | | FixesPatrick Wendell2014-01-082-2/+2
| | | | |
| * | | | Rename to ClientPatrick Wendell2014-01-072-8/+5
| | | | |
| * | | | Adding --verbose option to DriverClientPatrick Wendell2014-01-072-4/+22
| | | | |
| * | | | Adding unit tests and some refactoring to promote testability.Patrick Wendell2014-01-078-35/+251
| | | | |
| * | | | Fixes after mergePatrick Wendell2014-01-063-6/+8
| | | | |
| * | | | Merge remote-tracking branch 'apache-github/master' into standalone-driverPatrick Wendell2014-01-06123-1234/+2220
| |\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala core/src/main/scala/org/apache/spark/deploy/client/TestClient.scala core/src/main/scala/org/apache/spark/deploy/master/Master.scala core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
| * | | | | Changes based on review feedback.Patrick Wendell2014-01-067-22/+34
| | | | | |
| * | | | | Respect supervise option at MasterPatrick Wendell2013-12-291-3/+15
| | | | | |
| * | | | | Slight change to retry logicPatrick Wendell2013-12-291-2/+3
| | | | | |
| * | | | | TODO clean-upPatrick Wendell2013-12-295-6/+6
| | | | | |
| * | | | | Adding driver ID to submission responsePatrick Wendell2013-12-292-2/+2
| | | | | |
| * | | | | Documentation and adding supervise optionPatrick Wendell2013-12-294-13/+18
| | | | | |
| * | | | | Changes to allow fate sharing of drivers/executors and workers.Patrick Wendell2013-12-2917-133/+239
| | | | | |
| * | | | | Some notes and TODO about dependenciesPatrick Wendell2013-12-271-1/+7
| | | | | |
| * | | | | Intermediate clean-up of tests to appease jenkinsPatrick Wendell2013-12-261-10/+25
| | | | | |
| * | | | | Minor fixesPatrick Wendell2013-12-263-20/+25
| | | | | |
| * | | | | Addressing smaller changes from Aaron's reviewPatrick Wendell2013-12-267-27/+31
| | | | | |
| * | | | | Merge pull request #1 from aarondav/driverPatrick Wendell2013-12-261-62/+31
| |\ \ \ \ \ | | | | | | | | | | | | | | Refactor DriverClient to be more Actor-based
| | * | | | | Refactor DriverClient to be more Actor-basedAaron Davidson2013-12-251-62/+31
| | | | | | |
| * | | | | | Removing accidental filePatrick Wendell2013-12-262-10/+1
| | | | | | |
| * | | | | | Updated approach to driver restartingPatrick Wendell2013-12-262-23/+30
| |/ / / / /
| * | | | | Removing un-used variablePatrick Wendell2013-12-251-2/+0
| | | | | |
| * | | | | Small fix from rebasePatrick Wendell2013-12-251-1/+1
| | | | | |
| * | | | | Minor bug fixPatrick Wendell2013-12-252-1/+6
| | | | | |
| * | | | | Minor style clean-upPatrick Wendell2013-12-255-18/+16
| | | | | |
| * | | | | Import clean-up (yay Aaron)Patrick Wendell2013-12-2510-38/+33
| | | | | |
| * | | | | Adding scheduling and reporting based on coresPatrick Wendell2013-12-256-8/+14
| | | | | |
| * | | | | Adding better option parsingPatrick Wendell2013-12-259-42/+142
| | | | | |
| * | | | | Initial cut at driver submission.Patrick Wendell2013-12-2516-53/+781
| | | | | |
| * | | | | Renaming Client => AppClientPatrick Wendell2013-12-254-11/+12
| | | | | |
* | | | | | Merge pull request #372 from pwendell/log4j-fix-1Patrick Wendell2014-01-091-0/+1
|\ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | Send logs to stderr by default (instead of stdout).
| * | | | | | Send logs to stderr by default (instead of stdout).Patrick Wendell2014-01-091-0/+1
| | | | | | |
* | | | | | | Merge pull request #362 from mateiz/conf-gettersMatei Zaharia2014-01-0930-74/+73
|\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Use typed getters for configuration settings This improves some of the code style after SPARK-544.
| * | | | | | | Use typed getters for configuration settingsMatei Zaharia2014-01-0930-74/+73
| | | | | | | |
* | | | | | | | Merge pull request #361 from rxin/cleanReynold Xin2014-01-094-55/+68
|\ \ \ \ \ \ \ \ | |/ / / / / / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Minor style cleanup. Mostly on indenting & line width changes. Focused on the few important files since they are the files that new contributors usually read first.
| * | | | | | | Minor update on SparkContext.broadcast's JavaDoc.Reynold Xin2014-01-091-2/+2
| | | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-15 09:14:54 +0000